G2Easy★ Core Essential · Industrial BedrockPart G · Architectures & Parameter-Efficient Fine-Tuning

LoRA (Low-Rank Adaptation)

⚡Flashcard Drill➔

Freeze pretrained weights and learn low-rank bypass updates A*B that merge seamlessly for inference.

⏱️ Time: 前向仅增加极轻量 O(2 * B * S * D * r) 计算
💾 Space: 可训练参数量压缩 99% 以上(r=8 时仅为千分之几)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class LoRALinear:
def __init__(self, in_features: int, out_features: int, r: int = 8, alpha: float = 16.0):
self.in_features = in_features
self.out_features = out_features
self.r = r
self.scaling =
# 1. 原始冻结主权重
self.weight = np.random.randn(in_features, out_features) * 0.02
# 2. 低秩适配旁路参数
# A 采用高斯分布初始化,B 全零初始化确保初始 delta_W = 0
self.lora_A = np.random.randn(in_features, r) * (1.0 / np.sqrt(in_features))
self.lora_B = np.zeros((r, out_features))
def forward(self, x: np.ndarray) -> np.ndarray:
"""
x: (B, S, in_features)
"""
# 主路前向
base_out = x @ self.weight
# 旁路低秩前向: 先降维再升维,降低计算量
lora_out = (x @ self.lora_A) @ self.lora_B * self.scaling
return
def merge_weights(self):
"""生产部署时将 LoRA 权重折叠合并回主干"""
self.weight += (self.lora_A @ self.lora_B) * self.scaling
# 清空旁路
self.lora_A = None
self.lora_B = None
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does QLoRA fit 65B-scale model fine-tuning onto a single consumer GPU (e.g., 24GB/48GB)?
Next: MoE Top-K Routing & Auxiliary Load Balancing Loss →
💡

Core Mental Anchor / Mnemonic

Freeze W0, project through narrow rank r via A (random) and B (zero init), merge W0 + (alpha/r)BA at serving.

📐 Mathematical Derivation & Core Formula

h=W0x+ΔWx=W0x+αr(xA)B,A∈Rd×r,B∈Rr×kh = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} (x A) B, \quad A \in \mathbb{R}^{d \times r}, B \in \mathbb{R}^{r \times k}
### Intrinsic Rank Hypothesis & Zero-Latency Merge
LoRA (Hu et al., 2021) observes that task-adaptation weight updates ΔW\Delta W have low intrinsic rank. Rather than tuning full W0∈Rd×kW_0 \in \mathbb{R}^{d \times k}, it freezes W0W_0 and learns low-rank adapters ΔW=αrBA\Delta W = \frac{\alpha}{r} B A with A∼N(0,σ2)A \sim \mathcal{N}(0, \sigma^2) and B=0B = 0 (ensuring ΔW=0\Delta W = 0 at step 0).
At deployment, Wmerged=W0+αrBAW_{\text{merged}} = W_0 + \frac{\alpha}{r} B A folds back into the base weights with zero inference latency overhead.

🔄 Tensor Dimensions & Shape Flow

x: (B, S, D) -> main: x @ W + bypass: ((x @ A) @ B) * (alpha/r) -> (B, S, K)

🛡️ Numerical Stability & Pitfalls Checklist

  • Initialize B to all zeros so delta-W = 0 at step 0 and pretrained behavior starts intact.
  • Multiply (x @ A) @ B left-to-right to keep intermediate rank r small instead of materializing A @ B.
  • Scaling by alpha / r helps keep learning-rate tuning stable when changing rank r.

💻 Reference Code Implementation

import numpy as np

class LoRALinear:
    def __init__(self, in_features: int, out_features: int, r: int = 8, alpha: float = 16.0):
        self.in_features = in_features
        self.out_features = out_features
        self.r = r
        self.scaling = alpha / r
        
        # 1. 原始冻结主权重
        self.weight = np.random.randn(in_features, out_features) * 0.02
        
        # 2. 低秩适配旁路参数
        # A 采用高斯分布初始化,B 全零初始化确保初始 delta_W = 0
        self.lora_A = np.random.randn(in_features, r) * (1.0 / np.sqrt(in_features))
        self.lora_B = np.zeros((r, out_features))
        
    def forward(self, x: np.ndarray) -> np.ndarray:
        """
        x: (B, S, in_features)
        """
        # 主路前向
        base_out = x @ self.weight
        # 旁路低秩前向: 先降维再升维,降低计算量
        lora_out = (x @ self.lora_A) @ self.lora_B * self.scaling
        return base_out + lora_out
        
    def merge_weights(self):
        """生产部署时将 LoRA 权重折叠合并回主干"""
        self.weight += (self.lora_A @ self.lora_B) * self.scaling
        # 清空旁路
        self.lora_A = None
        self.lora_B = None

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
layer = LoRALinear(in_features=16, out_features=16, r=4, alpha=8.0)
x = np.random.randn(2, 4, 16)
# 初始阶段 B=0,输出应严格等于纯 base 权重输出
assert np.allclose(layer.forward(x), x @ layer.weight)
# 模拟 B 训练后有值
layer.lora_B = np.ones((4, 16)) * 0.1
out_lora = layer.forward(x)
# 合并权重后
layer.merge_weights()
out_merged = x @ layer.weight
assert np.allclose(out_lora, out_merged), "合并权重输出与原 LoRA 输出不一致!"
print("✓ LoRA 前向与权重合并自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How does QLoRA fit 65B-scale model fine-tuning onto a single consumer GPU (e.g., 24GB/48GB)?
QLoRA combines 4-bit NormalFloat (NF4) base weight quantization, double quantization of quantization constants, and paged CPU-offloaded optimizers to handle memory spikes.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Conv2d from Scratch with Stride & PaddingBack to Part G & All 105Next: MoE Top-K Routing & Auxiliary Load Balancing Loss →