⚡ LLMsID: attention-scaled-dot

Scaled Dot-Product Attention

Q: Why divide by $\sqrt{d_k}$? What breaks without it (variance analysis, softmax saturation)?

【Scaled Dot-Product Attention 核心考点 1】Why divide by $\sqrt{d_k}$? What breaks without it (variance analysis, softmax saturation)?。技术解析：Definition: Scaled Dot-Product Attention is the core attention operator of the Transformer: Attention(Q,K,V) = softmax(\frac{Q K^T}{\sqrt{d_k}}) V. Dividing by \sqrt{d_k} normalizes the dot-product variance: if Q and K have per-dimension variance 1, the variance of a d_k-dim dot ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/attention-scaled-dot

Q: Time and space complexity of scaled dot-product attention? Why is $O(n^2)$ the long-context bottleneck?

【Scaled Dot-Product Attention 核心考点 2】Time and space complexity of scaled dot-product attention? Why is $O(n^2)$ the long-context bottleneck?。技术解析：Definition: Scaled Dot-Product Attention is the core attention operator of the Transformer: Attention(Q,K,V) = softmax(\frac{Q K^T}{\sqrt{d_k}}) V. Dividing by \sqrt{d_k} normalizes the dot-product variance: if Q and K have per-dimension variance 1, the variance of a d_k-dim dot ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/attention-scaled-dot

Q: Shapes of the Q/K/V projections? How are heads split and concatenated?

【Scaled Dot-Product Attention 核心考点 3】Shapes of the Q/K/V projections? How are heads split and concatenated?。技术解析：Definition: Scaled Dot-Product Attention is the core attention operator of the Transformer: Attention(Q,K,V) = softmax(\frac{Q K^T}{\sqrt{d_k}}) V. Dividing by \sqrt{d_k} normalizes the dot-product variance: if Q and K have per-dimension variance 1, the variance of a d_k-dim dot ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/attention-scaled-dot

Q: Difference and implementation of padding mask vs causal mask?

【Scaled Dot-Product Attention 核心考点 4】Difference and implementation of padding mask vs causal mask?。技术解析：Definition: Scaled Dot-Product Attention is the core attention operator of the Transformer: Attention(Q,K,V) = softmax(\frac{Q K^T}{\sqrt{d_k}}) V. Dividing by \sqrt{d_k} normalizes the dot-product variance: if Q and K have per-dimension variance 1, the variance of a d_k-dim dot ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/attention-scaled-dot

Q: Single vs multi-head: why h heads rather than one wide $d$-dim head?

【Scaled Dot-Product Attention 核心考点 5】Single vs multi-head: why h heads rather than one wide $d$-dim head?。技术解析：Definition: Scaled Dot-Product Attention is the core attention operator of the Transformer: Attention(Q,K,V) = softmax(\frac{Q K^T}{\sqrt{d_k}}) V. Dividing by \sqrt{d_k} normalizes the dot-product variance: if Q and K have per-dimension variance 1, the variance of a d_k-dim dot ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/attention-scaled-dot

缩放点积注意力

🎯Core Definition

Scaled Dot-Product Attention is the core attention operator of the Transformer:

Attention(Q,K,V) = softmax(\frac{Q K^T}{\sqrt{d_k}}) V

. Dividing by

\sqrt{d_k}

normalizes the dot-product variance: if Q and K have per-dimension variance 1, the variance of a

d_k

-dim dot product grows linearly with

d_k

; without scaling, softmax inputs saturate as dimensionality grows (e.g.

d_k = 128

\sqrt{128} \approx 11.3

) and gradients vanish.

💡Use Cases

every head softly matches all tokens in the sequence — the foundation for MHA/GQA/MLA, FlashAttention and RoPE; interviews typically start with "why divide by

\sqrt{d_k}

" and then probe complexity, masks and head splitting.

⚡Key Problems Solved

unlike unscaled dot products, scaling keeps softmax entropy and gradients stable at any dimensionality so deep networks train reliably; but time and memory stay