M5-130M5: NLP & Large Language ModelsPEFT (LoRA / QLoRA / Prefix Tuning)Easy
⚡Flashcard Drill➔Mastery:
LoRA Foundations, Mathematical Formulation, and Hyperparameter Dynamics
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
LoRA freezes pre-trained base weights and injects trainable rank-decomposition matrices into transformer layers, drastically slashing trainable parameter counts and GPU memory while preserving full adaptation performance.
📌 Key Takeaways
- •Low-rank parameterization: represents weight delta as , with initialized from Gaussian noise and initialized to zero
- •Zero initial divergence: zero-initialization of guarantees that training begins exactly at the pre-trained base model state with
- •Hyperparameter mechanics: rank bounds adaptation subspace capacity, while scaling factor stabilizes learning rate dynamics across rank adjustments
📐 Mathematical Derivations
Mathematical Mechanism: 1. Intrinsic Dimensionality Hypothesis (Hu et al., 2021): Drawing on Aghajanyan et al., fine-tuning updates occupy a low intrinsic dimension. For a frozen pre-trained weight matrix , LoRA decomposes the adaptation increment into two low-rank matrices: where . For and , parameters shrink from to (a reduction). 2. Forward Pass Formulation: For input hidden state : Computing requires FLOPs, which is negligible compared to the base projection. 3. Initialization Protocol: Matrix is drawn from a Gaussian distribution or He uniform initialization, and matrix is initialized strictly to zero (). This guarantees: ensuring that fine-tuning starts smoothly from the pre-trained baseline without initial degradation. 4. Role of Scaling Factor $\alpha$: The scaling multiplier stabilizes optimizer dynamics. When experimenting with different ranks , keeping the ratio constant eliminates the need to recalibrate learning rates.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Target Module Selection: While the original paper targeted only attention projection weights (), modern consensus (Dettmers et al., QLoRA) proves that applying LoRA across all linear layers (, plus MLP gates ) at a lower rank ( or ) universally outperforms applying a higher rank () exclusively to attention layers. ② Optimizer Memory Slashed: In full fine-tuning, AdamW stores first and second moment states for all parameters ( bytes per parameter in FP32), consuming model size in optimizer states alone. By restricting gradients to low-rank matrices ( of parameters), optimizer state memory drops to near zero, unlocking single-GPU training of massive models. ③ Zero Inference Latency Penalty: Prior to production deployment, compute . Serving runs on standard optimized GEMM kernels without latency overhead. ④ Rank Over-Parameterization Risks: Setting unnecessarily high () increases vulnerability to overfitting on small instruction datasets and inflates checkpoint storage without improving generalization. ⑤ Interview Strategy: State the formula , articulate the asymmetric initialization (), explain the optimizer memory savings mechanism, and advocate for 'all linear modules at low rank'.
⚠️ Common Interview Pitfalls
- ✕Initializing matrix with random non-zero weights, destroying pre-trained representations and destabilizing early training
- ✕Targeting only attention query and value projections while ignoring feed-forward network (FFN) layers, constraining model capacity
- ✕Adjusting rank without scaling proportionally, inadvertently destabilizing the effective learning rate schedule
🎯 Interviewer Follow-up Inquiries
- ?Why is matrix initialized to zero while matrix is initialized to random Gaussian values rather than vice versa?
- ?Why does applying LoRA across all linear layers with rank outperform applying rank strictly to attention weights?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚