M4-014M4: Sequences & TransformersTransformer Architecture AnatomyEasy
⚡Flashcard Drill➔Mastery:
Scaled Dot-Product Attention Formula and the Vital Role of the 1/√d_k Factor
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
; dividing by normalizes logit variance to 1.0, preventing softmax saturation and gradient vanishing.
📌 Key Takeaways
- •Formula:
- •Variance dynamics: inner product of two independent zero-mean unit-variance vectors has variance
- •Softmax protection: without scaling, large pushes logits to , driving softmax into extreme saturation where gradients vanish
📐 Mathematical Derivations
Mathematical Proof of Variance Scaling (Vaswani et al., 2017):
Let query vector and key vector .
Assume components are independent random variables with zero mean and unit variance: .
The dot product is: .
Compute expectation and variance of the sum of independent terms:
.
.
The standard deviation of the unscaled dot product is .
- If , standard deviation is . Dot product values easily reach magnitudes .
- In softmax: . When , and .
- The derivative is . As , , causing backpropagated attention gradients to vanish completely.
Dividing by normalizes the variance: , keeping inputs in softmax's sensitive non-saturated gradient regime.
Let query vector and key vector .
Assume components are independent random variables with zero mean and unit variance: .
The dot product is: .
Compute expectation and variance of the sum of independent terms:
.
.
The standard deviation of the unscaled dot product is .
- If , standard deviation is . Dot product values easily reach magnitudes .
- In softmax: . When , and .
- The derivative is . As , , causing backpropagated attention gradients to vanish completely.
Dividing by normalizes the variance: , keeping inputs in softmax's sensitive non-saturated gradient regime.
🏭 Production Trade-offs
Head Dimension Constancy: In Transformer architectures, head dimension is typically kept constant (e.g., or ) while total model hidden dimension scales with head count , preserving attention scale stability.
⚠️ Common Interview Pitfalls
- ✕Scaling by or instead of , which over-flattens attention weights into an uninformative uniform distribution
- ✕Applying the division after the softmax operator instead of inside the logits
🎯 Interviewer Follow-up Inquiries
- ?Why does the dot product variance equal rather than ?
- ?How does QK-Norm relate to the scaling factor in deep foundation models?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚