G2Easy★ Core Essential · Industrial BedrockPart G · Architectures & Parameter-Efficient Fine-Tuning
LoRA (Low-Rank Adaptation)
Freeze pretrained weights and learn low-rank bypass updates A*B that merge seamlessly for inference.
⏱️ Time:
前向仅增加极轻量 O(2 * B * S * D * r) 计算💾 Space:
可训练参数量压缩 99% 以上(r=8 时仅为千分之几)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class LoRALinear:
def __init__(self, in_features: int, out_features: int, r: int = 8, alpha: float = 16.0):
self.in_features = in_features
self.out_features = out_features
self.r = r
self.scaling =
# 1. 原始冻结主权重
self.weight = np.random.randn(in_features, out_features) * 0.02
# 2. 低秩适配旁路参数
# A 采用高斯分布初始化,B 全零初始化确保初始 delta_W = 0
self.lora_A = np.random.randn(in_features, r) * (1.0 / np.sqrt(in_features))
self.lora_B = np.zeros((r, out_features))
def forward(self, x: np.ndarray) -> np.ndarray:
"""
x: (B, S, in_features)
"""
# 主路前向
base_out = x @ self.weight
# 旁路低秩前向: 先降维再升维,降低计算量
lora_out = (x @ self.lora_A) @ self.lora_B * self.scaling
return
def merge_weights(self):
"""生产部署时将 LoRA 权重折叠合并回主干"""
self.weight += (self.lora_A @ self.lora_B) * self.scaling
# 清空旁路
self.lora_A = None
self.lora_B = None
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does QLoRA fit 65B-scale model fine-tuning onto a single consumer GPU (e.g., 24GB/48GB)?
💡
Core Mental Anchor / Mnemonic
Freeze W0, project through narrow rank r via A (random) and B (zero init), merge W0 + (alpha/r)BA at serving.
📐 Mathematical Derivation & Core Formula
### Intrinsic Rank Hypothesis & Zero-Latency Merge
LoRA (Hu et al., 2021) observes that task-adaptation weight updates have low intrinsic rank. Rather than tuning full , it freezes and learns low-rank adapters with and (ensuring at step 0).
At deployment, folds back into the base weights with zero inference latency overhead.
LoRA (Hu et al., 2021) observes that task-adaptation weight updates have low intrinsic rank. Rather than tuning full , it freezes and learns low-rank adapters with and (ensuring at step 0).
At deployment, folds back into the base weights with zero inference latency overhead.
🔄 Tensor Dimensions & Shape Flow
x: (B, S, D) -> main: x @ W + bypass: ((x @ A) @ B) * (alpha/r) -> (B, S, K)
🛡️ Numerical Stability & Pitfalls Checklist
- Initialize B to all zeros so delta-W = 0 at step 0 and pretrained behavior starts intact.
- Multiply
(x @ A) @ Bleft-to-right to keep intermediate rank r small instead of materializingA @ B. - Scaling by
alpha / rhelps keep learning-rate tuning stable when changing rank r.
💻 Reference Code Implementation
import numpy as np
class LoRALinear:
def __init__(self, in_features: int, out_features: int, r: int = 8, alpha: float = 16.0):
self.in_features = in_features
self.out_features = out_features
self.r = r
self.scaling = alpha / r
# 1. 原始冻结主权重
self.weight = np.random.randn(in_features, out_features) * 0.02
# 2. 低秩适配旁路参数
# A 采用高斯分布初始化,B 全零初始化确保初始 delta_W = 0
self.lora_A = np.random.randn(in_features, r) * (1.0 / np.sqrt(in_features))
self.lora_B = np.zeros((r, out_features))
def forward(self, x: np.ndarray) -> np.ndarray:
"""
x: (B, S, in_features)
"""
# 主路前向
base_out = x @ self.weight
# 旁路低秩前向: 先降维再升维,降低计算量
lora_out = (x @ self.lora_A) @ self.lora_B * self.scaling
return base_out + lora_out
def merge_weights(self):
"""生产部署时将 LoRA 权重折叠合并回主干"""
self.weight += (self.lora_A @ self.lora_B) * self.scaling
# 清空旁路
self.lora_A = None
self.lora_B = None
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
layer = LoRALinear(in_features=16, out_features=16, r=4, alpha=8.0)
x = np.random.randn(2, 4, 16)
# 初始阶段 B=0,输出应严格等于纯 base 权重输出
assert np.allclose(layer.forward(x), x @ layer.weight)
# 模拟 B 训练后有值
layer.lora_B = np.ones((4, 16)) * 0.1
out_lora = layer.forward(x)
# 合并权重后
layer.merge_weights()
out_merged = x @ layer.weight
assert np.allclose(out_lora, out_merged), "合并权重输出与原 LoRA 输出不一致!"
print("✓ LoRA 前向与权重合并自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How does QLoRA fit 65B-scale model fine-tuning onto a single consumer GPU (e.g., 24GB/48GB)?
QLoRA combines 4-bit NormalFloat (
NF4) base weight quantization, double quantization of quantization constants, and paged CPU-offloaded optimizers to handle memory spikes.⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.