M5-130M5: NLP & Large Language ModelsPEFT (LoRA / QLoRA / Prefix Tuning)Easy
⚡Flashcard Drill➔Mastery:

LoRA Foundations, Mathematical Formulation, and Hyperparameter Dynamics

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-131 →
📐 Mathematical Definition
W′=W+αrBA,B∈Rd×r,A∈Rr×k;train A,B onlyW'=W+\frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},A\in\mathbb{R}^{r\times k};\qquad \text{train }A,B\ \text{only}
⚡ One-Sentence Core Answer
LoRA freezes pre-trained base weights and injects trainable rank-decomposition matrices into transformer layers, drastically slashing trainable parameter counts and GPU memory while preserving full adaptation performance.

📌 Key Takeaways

  • •
    Low-rank parameterization: represents weight delta as  ΔW=αrBA \,\Delta W = \frac{\alpha}{r} B A\,, with AA initialized from Gaussian noise and BB initialized to zero
  • •
    Zero initial divergence: zero-initialization of BB guarantees that training begins exactly at the pre-trained base model state with  ΔW=0 \,\Delta W = 0\,
  • •
    Hyperparameter mechanics: rank rr bounds adaptation subspace capacity, while scaling factor α\alpha stabilizes learning rate dynamics across rank adjustments

📐 Mathematical Derivations

Mathematical Mechanism: 1. Intrinsic Dimensionality Hypothesis (Hu et al., 2021): Drawing on Aghajanyan et al., fine-tuning updates occupy a low intrinsic dimension. For a frozen pre-trained weight matrix W0∈Rd×kW_0 \in \mathbb{R}^{d \times k}, LoRA decomposes the adaptation increment ΔW\Delta W into two low-rank matrices: W=W0+ΔW=W0+αrBA,B∈Rd×r,  A∈Rr×kW = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A, \quad B \in \mathbb{R}^{d \times r}, \; A \in \mathbb{R}^{r \times k} where r≪min⁡(d,k)r \ll \min(d, k). For d=k=4096d = k = 4096 and r=8r = 8, parameters shrink from 16.7M16.7\text{M} to 2×8×4096=65.5K2 \times 8 \times 4096 = 65.5\text{K} (a 256×256\times reduction). 2. Forward Pass Formulation: For input hidden state x∈Rb×dx \in \mathbb{R}^{b \times d}: h=xW0+αrx(ATBT)=xW0+αr(xAT)BTh = x W_0 + \frac{\alpha}{r} x (A^T B^T) = x W_0 + \frac{\alpha}{r} (x A^T) B^T Computing (xAT)BT(x A^T) B^T requires O(b⋅d⋅r+b⋅r⋅k)\mathcal{O}(b \cdot d \cdot r + b \cdot r \cdot k) FLOPs, which is negligible compared to the base projection. 3. Initialization Protocol: Matrix AA is drawn from a Gaussian distribution N(0,σ2)\mathcal{N}(0, \sigma^2) or He uniform initialization, and matrix BB is initialized strictly to zero (B=0B = 0). This guarantees: ΔW=αrBA=0  ⟹  W=W0at t=0\Delta W = \frac{\alpha}{r} B A = 0 \implies W = W_0 \quad \text{at } t = 0 ensuring that fine-tuning starts smoothly from the pre-trained baseline without initial degradation. 4. Role of Scaling Factor $\alpha$: The scaling multiplier αr\frac{\alpha}{r} stabilizes optimizer dynamics. When experimenting with different ranks rr, keeping the ratio αr\frac{\alpha}{r} constant eliminates the need to recalibrate learning rates.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Target Module Selection: While the original paper targeted only attention projection weights (Wq,WvW_q, W_v), modern consensus (Dettmers et al., QLoRA) proves that applying LoRA across all linear layers (Wq,Wk,Wv,WoW_q, W_k, W_v, W_o, plus MLP gates Wgate,Wup,WdownW_{\text{gate}}, W_{\text{up}}, W_{\text{down}}) at a lower rank (r=8r=8 or 1616) universally outperforms applying a higher rank (r=64r=64) exclusively to attention layers. ② Optimizer Memory Slashed: In full fine-tuning, AdamW stores first and second moment states for all parameters (88 bytes per parameter in FP32), consuming 4×4\times model size in optimizer states alone. By restricting gradients to low-rank matrices (<0.5%< 0.5\% of parameters), optimizer state memory drops to near zero, unlocking single-GPU training of massive models. ③ Zero Inference Latency Penalty: Prior to production deployment, compute Wmerged=W0+αrBAW_{\text{merged}} = W_0 + \frac{\alpha}{r} B A. Serving runs on standard optimized GEMM kernels without latency overhead. ④ Rank Over-Parameterization Risks: Setting rr unnecessarily high (r≥128r \ge 128) increases vulnerability to overfitting on small instruction datasets and inflates checkpoint storage without improving generalization. ⑤ Interview Strategy: State the formula W=W0+αrBAW = W_0 + \frac{\alpha}{r} B A, articulate the asymmetric initialization (A∼N,B=0A \sim \mathcal{N}, B = 0), explain the optimizer memory savings mechanism, and advocate for 'all linear modules at low rank'.
⚠️ Common Interview Pitfalls
  • ✕
    Initializing matrix BB with random non-zero weights, destroying pre-trained representations and destabilizing early training
  • ✕
    Targeting only attention query and value projections while ignoring feed-forward network (FFN) layers, constraining model capacity
  • ✕
    Adjusting rank rr without scaling α\alpha proportionally, inadvertently destabilizing the effective learning rate schedule
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is matrix BB initialized to zero while matrix AA is initialized to random Gaussian values rather than vice versa?
  • ?
    Why does applying LoRA across all linear layers with rank r=8r=8 outperform applying rank r=64r=64 strictly to attention weights?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-129: Evaluation Methodologies and Safety Risks in Model Merging📋 Back to BankNext →M5-131: The Three Pillars of QLoRA: NF4, Double Quantization, and Paged Optimizers