M5-136M5: NLP & Large Language ModelsPEFT (LoRA / QLoRA / Prefix Tuning)Hard
⚡Flashcard Drill➔Mastery:
LoRA Architectural Variants (DoRA, LoRA+, rsLoRA) and Rank Selection Strategies
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
DoRA decouples directional updates from magnitude scaling to mirror full fine-tuning learning patterns, LoRA+ optimizes convergence by setting asymmetric learning rates (), and rsLoRA stabilizes high-rank scaling via .
📌 Key Takeaways
- •DoRA (Weight-Decomposed Low-Rank Adaptation): decomposes weights into magnitude vector and directional matrix , applying LoRA strictly to direction to bridge the full fine-tuning capacity gap
- •LoRA+: sets a substantially higher learning rate for matrix than matrix (), correcting gradient flow imbalances and accelerating training convergence
- •rsLoRA (Rank-Stabilized LoRA): replaces standard scaling with , ensuring stable gradient dynamics and continuous capacity scaling when scaling to large ranks ()
📐 Mathematical Derivations
Mathematical Mechanism: 1. DoRA (Liu et al., 2024): Full fine-tuning alters weight magnitude and direction simultaneously with weak correlation, whereas standard LoRA shows a strong positive correlation between magnitude and directional shifts. DoRA decomposes weight matrix into magnitude vector and normalized directional matrix : where denotes column-wise vector norms. Vector is trained as a learnable parameter ( initially), and is updated via low-rank LoRA. Upon completion, DoRA merges back into a single matrix with zero inference latency. 2. LoRA+ (Hayou et al., 2024): Standard LoRA uses a uniform learning rate for both and . Theoretical analysis of infinite-width neural networks reveals that matrix features slow feature learning while matrix requires rapid adaptation. LoRA+ sets: Accelerating training convergence by up to and achieving higher benchmark accuracy at zero architectural or serving cost. 3. rsLoRA (Kalajdzievski, 2023): In standard LoRA, scaling factor causes gradient magnitudes to scale as , collapsing effective learning speed as rank expands. rsLoRA modifies the scaling factor to: proving that learning dynamics remain invariant to rank , enabling effective scaling to .
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Zero-Cost Nature of LoRA+: LoRA+ requires literally two lines of code in optimizer parameter group definitions (setting different learning rates for
lora_B and lora_A). It adds zero parameter overhead, zero memory penalty, and zero inference cost, making it an immediate best practice for production fine-tuning pipelines. ② DoRA Training Latency Overhead: While DoRA achieves performance remarkably close to full fine-tuning across vision and language benchmarks, computing column-wise norms during the forward and backward passes introduces a 15-25% training throughput penalty. When training compute is constrained, standard LoRA with LoRA+ is preferable. ③ Rank Selection Heuristics in Practice: (a) Style, QA, and Instruction Alignment: or is optimal; higher rank provides zero measurable gain and risks overfitting. (b) Complex Multi-Step Math and Code Synthesis: or captures complex logical step dependencies. (c) Domain Shift / Continuous Pre-training: to paired with rsLoRA (, e.g., ) prevents gradient collapse. ④ Mergeability Preservation: Both DoRA and rsLoRA retain exact weight-folding mergeability: prior to deployment, evaluate the final static matrix and deploy via standard vLLM or TensorRT-LLM engines. ⑤ Interview Strategy: Contrast magnitude vs direction in DoRA (), derive LoRA+'s asymmetric learning rate ratio , explain why rsLoRA's stabilizes gradient flow on large ranks, and outline rank selection rules of thumb ( for style, with rsLoRA for complex domains).⚠️ Common Interview Pitfalls
- ✕Scaling LoRA rank to without adopting rsLoRA scaling (), causing vanishing effective updates
- ✕Applying DoRA without accounting for the 20% training throughput slowdown caused by continuous column-wise normalization
- ✕Setting excessively large ranks () for simple instruction alignment, which increases memory footprint and triggers overfitting
🎯 Interviewer Follow-up Inquiries
- ?Why does decomposing weights into magnitude and direction in DoRA allow low-rank adaptation to closely approximate full fine-tuning?
- ?What mathematical imbalance in gradient updates between matrices A and B does LoRA+ resolve via asymmetric learning rates?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚