M4-018M4: Sequences & TransformersTransformer Architecture AnatomyHard
⚡Flashcard Drill➔Mastery:

Parameter and VRAM Memory Distribution in Transformer Architectures

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-019 →
📐 Mathematical Definition
params≈12d2L;FFN:Attn≈2:1;memtrain≈16Ψ+activations\text{params}\approx 12d^2L;\qquad \text{FFN}:\text{Attn}\approx 2:1;\qquad \text{mem}_{\text{train}}\approx 16\Psi+\text{activations}
⚡ One-Sentence Core Answer
Parameters are distributed 1:2 between Attention (4d24d^2) and MLP (8d28d^2); VRAM is dominated during training by optimizer states and activations, and during inference by the KV cache.

📌 Key Takeaways

  • •
    Layer parameters: Attention has 4d24d^2 parameters (WQ,WK,WV,WOW_Q, W_K, W_V, W_O); FFN has 8d28d^2 (standard) or 3×83d2=8d23 \times \frac{8}{3}d^2 = 8d^2 (SwiGLU)
  • •
    Training VRAM: Model Weights (2Ψ2\Psi in FP16), Gradients (2Ψ2\Psi), Adam States (12Ψ12\Psi), plus Activations (O(B⋅S⋅L⋅d)O(B \cdot S \cdot L \cdot d))
  • •
    Inference VRAM: Model Weights (2Ψ2\Psi in FP16 or 0.5Ψ0.5\Psi in INT4) plus dynamic KV Cache (2×2⋅B⋅S⋅nlayers⋅dkv2 \times 2 \cdot B \cdot S \cdot n_{\text{layers}} \cdot d_{\text{kv}} bytes)

📐 Mathematical Derivations

Exact Mathematical Derivations:
① Parameter Count per Layer (Hidden dimension $d$, Vocabulary $V$):
- Self-Attention: WQ,WK,WV,WOW_Q, W_K, W_V, W_O each of shape (d,d)  ⟹  4d2(d, d) \implies 4 d^2.
- SwiGLU MLP: Wgate,WupW_{\text{gate}}, W_{\text{up}} each (d,83d)(d, \frac{8}{3}d) and Wdown(83d,d)  ⟹  3×83d2=8d2W_{\text{down}} (\frac{8}{3}d, d) \implies 3 \times \frac{8}{3}d^2 = 8 d^2.
- Total per layer: 4d2+8d2=12d24 d^2 + 8 d^2 = 12 d^2.
- For LL layers: Ψ≈12Ld2+2Vd\Psi \approx 12 L d^2 + 2 V d (including input/output embeddings).
② Inference KV-Cache Memory Formula:
For each token in the sequence, we store Key and Value vectors across all layers.
For batch size BB, sequence length SS, layer count LL, and KV channels dkv=nkv_heads×dkd_{\text{kv}} = n_{\text{kv\_heads}} \times d_k (in FP16, 2 bytes/element):
MemoryKV=2×[2×B×S×L×dkv]=4 B S L dkv bytes\text{Memory}_{\text{KV}} = 2 \times [2 \times B \times S \times L \times d_{\text{kv}}] = 4 \, B \, S \, L \, d_{\text{kv}} \text{ bytes}.
Example: LLaMA-70B (L=80,dkv=1024L=80, d_{\text{kv}}=1024) with B=1B=1 and context S=128,000S=128,000 tokens:
MemoryKV=4×1×128000×80×1024≈41.94 GB\text{Memory}_{\text{KV}} = 4 \times 1 \times 128000 \times 80 \times 1024 \approx 41.94 \text{ GB} of VRAM just for the KV cache!

🏭 Production Trade-offs

Architecture optimization: The massive memory footprint of KV caches led directly to the universal adoption of Grouped-Query Attention (GQA), which shares key-value heads across multiple query heads, cutting KV-cache memory by 8×8\times.
⚠️ Common Interview Pitfalls
  • ✕
    Calculating inference memory solely based on model weight parameters, completely forgetting the massive KV cache footprint in long contexts
  • ✕
    Assuming SwiGLU requires more parameters than standard MLPs; intermediate dimension scaling preserves exact 8d28d^2 parity
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Grouped-Query Attention (GQA) reduce KV-cache memory bandwidth by factor 8×8\times in LLaMA-3?
  • ?
    What proportion of training memory is occupied by activations versus optimizer states in 70B models?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-017: Theoretical Expressive Power and Computational Limits of Transformers📋 Back to BankNext →M4-019: Normalization Architecture Choices in Transformers: Pre-LN, Post-LN, and RMSNorm