M4-023M4: Sequences & TransformersTransformer Architecture AnatomyEasy
⚡Flashcard Drill➔Mastery:

Attention Masking Mechanisms: Causal, Padding, and Prefix-LM Masks

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-024 →
📐 Mathematical Definition
mask: logits←logits+M,Mij={0allowed−∞masked\mathrm{mask}:\ \text{logits}\leftarrow\text{logits}+M,\quad M_{ij}=\begin{cases}0&\text{allowed}\\ -\infty&\text{masked}\end{cases}
⚡ One-Sentence Core Answer
Masks set attention logits of invalid or prohibited tokens to −∞-\infty prior to softmax, implementing causal autoregression, padding isolation, or hybrid prefix-LM dynamics.

📌 Key Takeaways

  • •
    Implementation: A=softmax(QKTd+M)VA = \text{softmax}\left(\frac{Q K^T}{\sqrt{d}} + M\right) V, where Mij=0M_{ij} = 0 for allowed tokens and −∞-\infty (or −109-10^9) for prohibited tokens
  • •
    Causal Mask: lower-triangular binary matrix (Mij=−∞M_{ij} = -\infty for j>ij > i), preventing tokens from looking into the future
  • •
    Padding Mask: sets Mij=−∞M_{ij} = -\infty wherever key token jj is a <pad> padding token
  • •
    Prefix-LM Mask: fully bidirectional within the prompt prefix, strictly causal within generated suffix tokens

📐 Mathematical Derivations

Mathematical Formulations:
① Causal Mask (Autoregressive Mask):
Enforces temporal causality: token ii can only attend to tokens j≤ij \le i.
Mcausal[i,j]={0if j≤i−∞if j>iM_{\text{causal}}[i, j] = \begin{cases} 0 & \text{if } j \le i \\ -\infty & \text{if } j > i \end{cases}.
In floating-point hardware (FP16/BF16), −∞-\infty is implemented as −109-10^9 or −65504-65504. In softmax: e−∞=0e^{-\infty} = 0. Prohibited future tokens receive exact zero attention weight: Pij=0P_{ij} = 0.
② Padding Mask:
When batching sequences of variable length, shorter sequences are padded with <pad> tokens.
Mpad[i,j]={0if token j≠<pad>−∞if token j=<pad>M_{\text{pad}}[i, j] = \begin{cases} 0 & \text{if token } j \ne \text{<pad>} \\ -\infty & \text{if token } j = \text{<pad>} \end{cases}.
Prevents dummy padding tokens from polluting context representations.
③ Prefix-LM Mask (Non-causal Prefix + Causal Generation):
Let prompt prefix tokens have length PP, and generation tokens have length GG.
Mprefix[i,j]={0if i≤P and j≤P (bidirectional prompt)0if i>P and j≤i (causal response)−∞otherwiseM_{\text{prefix}}[i, j] = \begin{cases} 0 & \text{if } i \le P \text{ and } j \le P \text{ (bidirectional prompt)} \\ 0 & \text{if } i > P \text{ and } j \le i \text{ (causal response)} \\ -\infty & \text{otherwise} \end{cases}.
Combines the bidirectional comprehension quality of BERT with the autoregressive generation capacity of GPT.

🏭 Production Trade-offs

Hardware optimization: In modern fused attention kernels (FlashAttention-2), causal masking is computed in hardware via thread block row-column index comparison (j>ij > i) without ever materializing an actual N×NN \times N mask matrix in GPU memory.
⚠️ Common Interview Pitfalls
  • ✕
    Using 00 as a mask value instead of −∞-\infty; setting logits to 00 results in equal non-zero attention weights (e0=1e^0 = 1)
  • ✕
    Encountering all-masked rows where an entire sequence is padding, causing softmax(−∞)=0/0=NaN\text{softmax}(-\infty) = 0/0 = \text{NaN}
🎯 Interviewer Follow-up Inquiries
  • ?
    How does FlashAttention implement causal masking in CUDA without creating an actual N×NN \times N mask tensor in memory?
  • ?
    What causes all-masked row NaNs in multi-head attention when using padding masks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-022: Depth vs Width Trade-offs in Transformers: Scaling Laws and Optimal Allocations📋 Back to BankNext →M4-024: Transformer Training Stability Techniques: QK-Norm, Logit Soft-Capping, and Auxiliary Z-Loss