M4-037M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Medium
⚡Flashcard Drill➔Mastery:
Sparse Attention Mechanisms and Sliding Window Attention (SWA)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Sparse attention restricts tokens to attending to local sliding windows, dilated strides, or global anchor tokens, cutting complexity to linear time.
📌 Key Takeaways
- •Sliding Window Attention (SWA / Mistral): token attends only to tokens within ; reduces complexity from to
- •Receptive field accumulation: stacking layers expands effective receptive field to through recursive multi-hop information routing
- •Global tokens: BigBird and Longformer add full bidirectional attention to special anchor tokens (e.g.,
[CLS]) to bridge global context
📐 Mathematical Derivations
Mathematical Formulations:
① Sliding Window Attention (Beltagy et al., Longformer; Jiang et al., Mistral 7B):
Let window size be . Attention mask is defined as:
.
- Complexity: Memory and FLOPs scale as , strictly linear in sequence length .
- Theoretical Multi-Hop Receptive Field:
At layer 1, token sees .
At layer 2, tokens inside have already attended to their own predecessors up to .
After stacked Transformer layers, the effective theoretical receptive field expands to: .
For Mistral (), the theoretical receptive field covers tokens!
② Rolling Buffer KV Cache:
During inference, cache size is capped at . Position overwrites position in circular buffers, capping VRAM memory permanently.
① Sliding Window Attention (Beltagy et al., Longformer; Jiang et al., Mistral 7B):
Let window size be . Attention mask is defined as:
.
- Complexity: Memory and FLOPs scale as , strictly linear in sequence length .
- Theoretical Multi-Hop Receptive Field:
At layer 1, token sees .
At layer 2, tokens inside have already attended to their own predecessors up to .
After stacked Transformer layers, the effective theoretical receptive field expands to: .
For Mistral (), the theoretical receptive field covers tokens!
② Rolling Buffer KV Cache:
During inference, cache size is capped at . Position overwrites position in circular buffers, capping VRAM memory permanently.
🏭 Production Trade-offs
GPU Hardware Reality: Irregular or dynamic sparsity patterns achieve zero hardware speedup on GPUs due to non-coalesced memory access. Sliding Window Attention succeeds because fixed, regular band-diagonal matrices are easily accelerated via custom FlashAttention kernels.
⚠️ Common Interview Pitfalls
- ✕Using pure Sliding Window Attention without global tokens or interleaved full-attention layers on retrieval tasks, causing complete loss of long-range single-hop recall
- ✕Confusing theoretical multi-hop receptive field () with effective single-hop attention capability
🎯 Interviewer Follow-up Inquiries
- ?How does Mistral's Rolling Buffer Cache cap inference KV-cache memory to a fixed window size ?
- ?Why do arbitrary graph-based sparse attention patterns fail to accelerate execution on modern GPU architectures?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚