M4-018M4: Sequences & TransformersTransformer Architecture AnatomyHard
⚡Flashcard Drill➔Mastery:
Parameter and VRAM Memory Distribution in Transformer Architectures
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Parameters are distributed 1:2 between Attention () and MLP (); VRAM is dominated during training by optimizer states and activations, and during inference by the KV cache.
📌 Key Takeaways
- •Layer parameters: Attention has parameters (); FFN has (standard) or (SwiGLU)
- •Training VRAM: Model Weights ( in FP16), Gradients (), Adam States (), plus Activations ()
- •Inference VRAM: Model Weights ( in FP16 or in INT4) plus dynamic KV Cache ( bytes)
📐 Mathematical Derivations
Exact Mathematical Derivations:
① Parameter Count per Layer (Hidden dimension $d$, Vocabulary $V$):
- Self-Attention: each of shape .
- SwiGLU MLP: each and .
- Total per layer: .
- For layers: (including input/output embeddings).
② Inference KV-Cache Memory Formula:
For each token in the sequence, we store Key and Value vectors across all layers.
For batch size , sequence length , layer count , and KV channels (in FP16, 2 bytes/element):
.
Example: LLaMA-70B () with and context tokens:
of VRAM just for the KV cache!
① Parameter Count per Layer (Hidden dimension $d$, Vocabulary $V$):
- Self-Attention: each of shape .
- SwiGLU MLP: each and .
- Total per layer: .
- For layers: (including input/output embeddings).
② Inference KV-Cache Memory Formula:
For each token in the sequence, we store Key and Value vectors across all layers.
For batch size , sequence length , layer count , and KV channels (in FP16, 2 bytes/element):
.
Example: LLaMA-70B () with and context tokens:
of VRAM just for the KV cache!
🏭 Production Trade-offs
Architecture optimization: The massive memory footprint of KV caches led directly to the universal adoption of Grouped-Query Attention (GQA), which shares key-value heads across multiple query heads, cutting KV-cache memory by .
⚠️ Common Interview Pitfalls
- ✕Calculating inference memory solely based on model weight parameters, completely forgetting the massive KV cache footprint in long contexts
- ✕Assuming SwiGLU requires more parameters than standard MLPs; intermediate dimension scaling preserves exact parity
🎯 Interviewer Follow-up Inquiries
- ?How does Grouped-Query Attention (GQA) reduce KV-cache memory bandwidth by factor in LLaMA-3?
- ?What proportion of training memory is occupied by activations versus optimizer states in 70B models?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚