M4-023M4: Sequences & TransformersTransformer Architecture AnatomyEasy
⚡Flashcard Drill➔Mastery:
Attention Masking Mechanisms: Causal, Padding, and Prefix-LM Masks
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Masks set attention logits of invalid or prohibited tokens to prior to softmax, implementing causal autoregression, padding isolation, or hybrid prefix-LM dynamics.
📌 Key Takeaways
- •Implementation: , where for allowed tokens and (or ) for prohibited tokens
- •Causal Mask: lower-triangular binary matrix ( for ), preventing tokens from looking into the future
- •Padding Mask: sets wherever key token is a
<pad>padding token - •Prefix-LM Mask: fully bidirectional within the prompt prefix, strictly causal within generated suffix tokens
📐 Mathematical Derivations
Mathematical Formulations:
① Causal Mask (Autoregressive Mask):
Enforces temporal causality: token can only attend to tokens .
.
In floating-point hardware (FP16/BF16), is implemented as or . In softmax: . Prohibited future tokens receive exact zero attention weight: .
② Padding Mask:
When batching sequences of variable length, shorter sequences are padded with
.
Prevents dummy padding tokens from polluting context representations.
③ Prefix-LM Mask (Non-causal Prefix + Causal Generation):
Let prompt prefix tokens have length , and generation tokens have length .
.
Combines the bidirectional comprehension quality of BERT with the autoregressive generation capacity of GPT.
① Causal Mask (Autoregressive Mask):
Enforces temporal causality: token can only attend to tokens .
.
In floating-point hardware (FP16/BF16), is implemented as or . In softmax: . Prohibited future tokens receive exact zero attention weight: .
② Padding Mask:
When batching sequences of variable length, shorter sequences are padded with
<pad> tokens..
Prevents dummy padding tokens from polluting context representations.
③ Prefix-LM Mask (Non-causal Prefix + Causal Generation):
Let prompt prefix tokens have length , and generation tokens have length .
.
Combines the bidirectional comprehension quality of BERT with the autoregressive generation capacity of GPT.
🏭 Production Trade-offs
Hardware optimization: In modern fused attention kernels (FlashAttention-2), causal masking is computed in hardware via thread block row-column index comparison () without ever materializing an actual mask matrix in GPU memory.
⚠️ Common Interview Pitfalls
- ✕Using as a mask value instead of ; setting logits to results in equal non-zero attention weights ()
- ✕Encountering all-masked rows where an entire sequence is padding, causing
🎯 Interviewer Follow-up Inquiries
- ?How does FlashAttention implement causal masking in CUDA without creating an actual mask tensor in memory?
- ?What causes all-masked row NaNs in multi-head attention when using padding masks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚