LoRA / QLoRA

Frozen W. Train A,B. Rank is the budget.

play

W' = W + α B A · frozen W · train A,B only

rank r 1α 1.0Δ params — vs full —

Adam only sees A and B. Frozen W stays INT4.

W′ = W + α B A

Toy 64×64 rank 4: 512 trainable vs 4096 frozen (11%). MSE 0.207→0.144. Adam only sees A,B.