M4-045M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Hard
⚡Flashcard Drill➔Mastery:
Alternatives to Softmax Attention: Sigmoid Attention and Normalization Variants
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Sigmoid attention decouples token competition and eliminates attention sinks; Softmax1 adds a dummy constant to the denominator to allow complete non-attention.
📌 Key Takeaways
- •Sigmoid Attention (Ramapuram et al., Apple 2024): ; independent binary gating per token
- •Softmax1 (Miller, 2023): ; allows all attention weights to sum to , eliminating forced attention sinks
- •Hardware advantage: Sigmoid attention computes element-wise without global reduction, enabling fast FlashAttention-style streaming
📐 Mathematical Derivations
Mathematical Formulations:
① Softmax Attention Limitations:
Softmax enforces global competition: . This has two flaws: (1) Forces attention sinks when no token is relevant; (2) Requires global sum reduction across all tokens, hindering streaming kernels.
② Sigmoid Attention:
.
Output: (or unnormalized ).
- Decoupled Weights: Every key token's relevance is evaluated independently. A query can attend to 0 tokens, 5 tokens, or all tokens with equal strength.
③ Softmax1 (Adding a Constant Sink):
.
If all logits , then , and all output weights . The model can choose to attend to nothing without being forced to dump weights onto the first token.
① Softmax Attention Limitations:
Softmax enforces global competition: . This has two flaws: (1) Forces attention sinks when no token is relevant; (2) Requires global sum reduction across all tokens, hindering streaming kernels.
② Sigmoid Attention:
.
Output: (or unnormalized ).
- Decoupled Weights: Every key token's relevance is evaluated independently. A query can attend to 0 tokens, 5 tokens, or all tokens with equal strength.
③ Softmax1 (Adding a Constant Sink):
.
If all logits , then , and all output weights . The model can choose to attend to nothing without being forced to dump weights onto the first token.
🏭 Production Trade-offs
FlashSigmoid Execution: Because Sigmoid has no global partition function , tokens can be processed in parallel across CUDA warps with zero inter-thread reduction synchronization, achieving theoretical speedups.
⚠️ Common Interview Pitfalls
- ✕Using Sigmoid attention without a negative bias (e.g., ), causing dense near-1 activations across all tokens
- ✕Swapping Softmax for Sigmoid in a pretrained model without full pretraining from scratch
🎯 Interviewer Follow-up Inquiries
- ?Why does Softmax1 mathematically prevent the formation of attention sinks on the BOS token?
- ?What computational speedup does Sigmoid attention offer over Softmax in distributed hardware kernels?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚