M4-042M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Hard
⚡Flashcard Drill➔Mastery:
Native Sparse Attention (NSA): Multi-Branch Hardware-Aligned Sparse Design
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
NSA combines token compression, top- block selection, and local sliding windows into three parallel branches, achieving hardware-aligned linear acceleration on long contexts.
📌 Key Takeaways
- •Hardware alignment: aligns all sparse lookups to contiguous memory blocks (e.g., tokens) to saturate GPU memory bandwidth
- •Tri-branch architecture: Compressed Branch (global coarse context) + Selected Branch (fine-grained top- blocks) + Window Branch (local syntax)
- •Native training: trains end-to-end with native sparsity, eliminating train-test distribution mismatch
📐 Mathematical Derivations
Architectural Anatomy (DeepSeek-AI, 2025; Native Sparse Attention):
Standard theoretical sparse attention (e.g., token-level routing) fails to accelerate on GPUs due to scattered memory access patterns. NSA enforces hardware-aligned block sparsity.
For query , attention is decomposed across three specialized branches:
① Compressed Token Branch (Global Coarse Context):
Compresses blocks of keys and values of size (e.g., 16 tokens) into a single representative vector using a lightweight 1D convolution: . Query attends globally across all compressed blocks with complexity, maintaining full-sequence awareness.
② Selected Block Branch (Fine-grained Detail):
Computes coarse block similarity scores between query and compressed keys . Selects the top- most relevant blocks (e.g., blocks of size 64). Loads only these specific continuous blocks into SRAM to compute exact token-level attention.
③ Sliding Window Branch (Local Syntax):
Computes standard local attention over the immediate preceding window (e.g., 512 tokens), capturing precise local word order and syntax.
Outputs from all three branches are gated and summed: .
Standard theoretical sparse attention (e.g., token-level routing) fails to accelerate on GPUs due to scattered memory access patterns. NSA enforces hardware-aligned block sparsity.
For query , attention is decomposed across three specialized branches:
① Compressed Token Branch (Global Coarse Context):
Compresses blocks of keys and values of size (e.g., 16 tokens) into a single representative vector using a lightweight 1D convolution: . Query attends globally across all compressed blocks with complexity, maintaining full-sequence awareness.
② Selected Block Branch (Fine-grained Detail):
Computes coarse block similarity scores between query and compressed keys . Selects the top- most relevant blocks (e.g., blocks of size 64). Loads only these specific continuous blocks into SRAM to compute exact token-level attention.
③ Sliding Window Branch (Local Syntax):
Computes standard local attention over the immediate preceding window (e.g., 512 tokens), capturing precise local word order and syntax.
Outputs from all three branches are gated and summed: .
🏭 Production Trade-offs
Native Sparse vs Post-hoc Pruning: Most sparse systems train dense models and prune them during inference, causing catastrophic distribution mismatch. NSA trains with sparse branches from step 0, ensuring optimal parameter adaptation.
⚠️ Common Interview Pitfalls
- ✕Implementing token-level fine-grained sparsity without block alignment, resulting in slower execution than dense FlashAttention
- ✕Omitting the local sliding window branch, which impairs fine-grained code parsing and syntax
🎯 Interviewer Follow-up Inquiries
- ?Why must sparse attention patterns be aligned to memory blocks (e.g., 64 tokens) to achieve speedups on GPUs?
- ?How does NSA select top- blocks dynamically without materializing the full attention matrix?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚