Asked about The linear algebra inside a language model
Why does LoRA use so few trainable parameters?
An answer that works
It represents the update to a weight matrix as the product of two thin matrices of rank r. For a d by d matrix that is 2 times d times r numbers instead of d squared, a small fraction when r is small. It assumes the useful update is low rank.
Reading an answer is not being able to give one. The section that teaches this is The linear algebra inside a language model, 3 hours of Linear algebra.
Learn this properlyAlso asked about Linear algebra
What does a dot product tell you?Why do matrix multiply shapes have to line up?What is broadcasting and why is it risky?What is the difference between L1 and L2 regularisation?What does SVD give you?How does PCA relate to eigenvectors?What does the dot product of two vectors tell you?How is linear regression a linear algebra problem?
All 10 questions on Linear algebra