LoRA/QLoRA VRAM & Weight Merge

LoRA/QLoRA Parameter-Efficient Fine-Tuning (PEFT), VRAM Budgeting & Adapter Lifecycle Management empowers engineers to fine-tune 7B-70B models on single commodity GPUs (e.g. RTX 4090 or A10G) and serve them with zero latency penalty; core components include: 1) LoRA (Low-Rank Adaptation): freezing base weights $W_0 \in \mathbb{R}^{d \times k}$ and injecting trainable low-rank decomposition matrices $\Delta W = \frac{\alpha}{r} (B \cdot A)$ ($A$ initialized with Gaussian, $B=0$, $r \ll d$), training $<1\%$ of total parameters and slashing optimizer memory by $>80\%$; 2) QLoRA: quantizing frozen base models into NF4 (NormalFloat4) 4-bit representations alongside Double Quantization and Paged Optimizers, enabling 70B fine-tuning on a single 24GB card; 3) Weight Merging & Multi-Tenancy: fusing adapter deltas via $W_{\text{final}} = W_0 + \frac{\alpha}{r} BA$ for zero-overhead deployment, or using S-LoRA/Punica for multi-tenant dynamic adapter hot-swapping.

Cost-effective domain LLM fine-tuning, multi-tenant vertical adapter serving, and zero-downtime hot upgrades.

Full fine-tuning 70B models demands 8x A100 GPU clusters and 600GB+ VRAM; LoRA/QLoRA delivers 99% of full fine-tuning performance at 1/10th the hardware budget.

Derive LoRA's forward pass $h = W_0 x + \frac{\alpha}{r} B A x$ and explain why $B$ is initialized to exact zero while $A$ is Gaussian?

Explain the relationship between Rank $r$ and scaling factor $\alpha$ (why $\alpha = 2r$ is common practice) in learning stability?

Compare applying LoRA solely on attention projections vs all linear FFN layers in model accuracy and parameter efficiency?

Why does QLoRA's NF4 (NormalFloat4) preserve information entropy better than uniform INT4 on normally distributed weights?

How does `merge_and_unload()` fuse LoRA weights into base tensors to achieve zero additional serving latency in vLLM?