M4-045M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Hard
⚡Flashcard Drill➔Mastery:

Alternatives to Softmax Attention: Sigmoid Attention and Normalization Variants

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-046 →
📐 Mathematical Definition
sigmoid attn: αij=σ(qi⋅kj/d);no row normalization\text{sigmoid attn}:\ \alpha_{ij}=\sigma(q_i\cdot k_j/\sqrt d);\qquad \text{no row normalization}
⚡ One-Sentence Core Answer
Sigmoid attention decouples token competition and eliminates attention sinks; Softmax1 adds a dummy constant to the denominator to allow complete non-attention.

📌 Key Takeaways

  • •
    Sigmoid Attention (Ramapuram et al., Apple 2024): Aij=σ(qiTkj/d−b)A_{ij} = \sigma(q_i^T k_j / \sqrt{d} - b); independent binary gating per token
  • •
    Softmax1 (Miller, 2023): ezi1+∑ezj\frac{e^{z_i}}{1 + \sum e^{z_j}}; allows all attention weights to sum to <1.0< 1.0, eliminating forced attention sinks
  • •
    Hardware advantage: Sigmoid attention computes element-wise without global reduction, enabling fast FlashAttention-style streaming

📐 Mathematical Derivations

Mathematical Formulations:
① Softmax Attention Limitations:
Softmax enforces global competition: ∑j=1NAij=1\sum_{j=1}^N A_{ij} = 1. This has two flaws: (1) Forces attention sinks when no token is relevant; (2) Requires global sum reduction across all tokens, hindering streaming kernels.
② Sigmoid Attention:
Aij=σ(qiTkjd+b)=11+exp⁡(−qiTkjd−b)A_{ij} = \sigma\left( \frac{q_i^T k_j}{\sqrt{d}} + b \right) = \frac{1}{1 + \exp\left( -\frac{q_i^T k_j}{\sqrt{d}} - b \right)}.
Output: Oi=1∑jAij+ϵ∑j=1NAijvjO_i = \frac{1}{\sum_j A_{ij} + \epsilon} \sum_{j=1}^N A_{ij} v_j (or unnormalized Oi=1N∑AijvjO_i = \frac{1}{N} \sum A_{ij} v_j).
- Decoupled Weights: Every key token's relevance is evaluated independently. A query can attend to 0 tokens, 5 tokens, or all tokens with equal strength.
③ Softmax1 (Adding a Constant Sink):
Softmax1(z)i=ezie0+∑j=1Nezj=ezi1+∑j=1Nezj\text{Softmax1}(z)_i = \frac{e^{z_i}}{e^0 + \sum_{j=1}^N e^{z_j}} = \frac{e^{z_i}}{1 + \sum_{j=1}^N e^{z_j}}.
If all logits zj≪0z_j \ll 0, then ∑ezj→0\sum e^{z_j} \to 0, and all output weights pi→0p_i \to 0. The model can choose to attend to nothing without being forced to dump weights onto the first token.

🏭 Production Trade-offs

FlashSigmoid Execution: Because Sigmoid has no global partition function ∑ezj\sum e^{z_j}, tokens can be processed in parallel across CUDA warps with zero inter-thread reduction synchronization, achieving theoretical speedups.
⚠️ Common Interview Pitfalls
  • ✕
    Using Sigmoid attention without a negative bias bb (e.g., b=−log⁡(N)b = -\log(N)), causing dense near-1 activations across all tokens
  • ✕
    Swapping Softmax for Sigmoid in a pretrained model without full pretraining from scratch
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Softmax1 mathematically prevent the formation of attention sinks on the BOS token?
  • ?
    What computational speedup does Sigmoid attention offer over Softmax in distributed hardware kernels?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-044: Attention Entropy Collapse and How QK-Norm Resolves It📋 Back to BankNext →M4-046: Fusing Attention with External Memory and Retrieval: kNN-LM and RETRO