D1Easy★ Core Essential · Industrial BedrockPart D · Attention Mechanisms & Transformer Blocks

Scaled Dot-Product Attention with Causal Mask

⚡Flashcard Drill➔

Scale dot products by 1/sqrt(d_k) to stabilize Softmax variance and apply a causal mask to block future tokens.

⏱️ Time: O(B * H * S_q * S_k * D) 经典二次复杂度
💾 Space: O(B * H * S_q * S_k) 显存瓶颈所在
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def scaled_dot_product_attention(
q: np.ndarray,
k: np.ndarray,
v: np.ndarray,
is_causal: bool = True
) -> np.ndarray:
"""
参数:
q: (B, H, S_q, D)
k: (B, H, S_k, D)
v: (B, H, S_k, D)
is_causal: 是否施加因果掩码 (S_q 必须等于 S_k)
返回:
out: (B, H, S_q, D)
"""
d_k = q.shape[-1]
# 1. 计算缩放点积分数 (B, H, S_q, D) @ (B, H, D, S_k) -> (B, H, S_q, S_k)
scores =
# 2. 注入因果掩码 (若需要)
if is_causal:
seq_len_q = q.shape[-2]
seq_len_k = k.shape[-2]
# 上三角为 True (未来信息需要被屏蔽)
mask = np.triu(np.ones((seq_len_q, seq_len_k), dtype=bool), k=1)
# 将被遮蔽区域赋为负无穷大
scores = np.where(mask, -1e9, scores)
# 3. 数值稳定 Softmax (减 max 归一化)
scores_max = np.max(scores, axis=-1, keepdims=True)
exp_scores = np.exp(scores - scores_max)
attn_weights = exp_scores / np.sum(exp_scores, axis=-1, keepdims=True)
# 4. 加权求和 (B, H, S_q, S_k) @ (B, H, S_k, D) -> (B, H, S_q, D)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: In FP16 mixed-precision training, why should you avoid masking with -1e30 or float('-inf')?
Next: Multi-Head Attention (MHA) →
💡

Core Mental Anchor / Mnemonic

Scale by 1/sqrt(d_k), mask strict upper triangle (k=1), Softmax, then weight Values.

📐 Mathematical Derivation & Core Formula

Attention(Q,K,V)=softmax(QK⊤dk+M)V\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right) V
### Variance Scaling & Causal Lower-Triangular Mask
Suppose components of qq and kk are i.i.d. zero-mean, unit-variance random variables. Their dot product q⋅k=∑i=1dkqikiq \cdot k = \sum_{i=1}^{d_k} q_i k_i has mean 00 and variance dkd_k.
When dkd_k is large (e.g., 64 or 128), large dot-product magnitudes push Softmax into saturated regions with near-zero gradients. Dividing by dk\sqrt{d_k} restores unit variance.
Causal mask: In autoregressive decoding, token ii cannot attend to j>ij > i. Upper-triangular entries (j>ij > i) receive a large negative mask (e.g., -1e9 or -1e4), driving emask→0e^{\text{mask}} \to 0 after Softmax.

🔄 Tensor Dimensions & Shape Flow

(B, H, Sq, D) @ (B, H, D, Sk) -> (B, H, Sq, Sk) -> scale + mask -> softmax -> @ V -> (B, H, Sq, D)

🛡️ Numerical Stability & Pitfalls Checklist

  • Use upper-triangular offset k=1 (np.triu(..., k=1)) so each token still attends to itself (k=0).
  • Fill masked positions with large negative finite values (e.g. -1e9 in FP32, -1e4 in FP16), never NaN.
  • Divide by sqrt(d_k) before applying masks and Softmax.

💻 Reference Code Implementation

import numpy as np

def scaled_dot_product_attention(
    q: np.ndarray,
    k: np.ndarray,
    v: np.ndarray,
    is_causal: bool = True
) -> np.ndarray:
    """
    参数:
        q: (B, H, S_q, D)
        k: (B, H, S_k, D)
        v: (B, H, S_k, D)
        is_causal: 是否施加因果掩码 (S_q 必须等于 S_k)
    返回:
        out: (B, H, S_q, D)
    """
    d_k = q.shape[-1]
    # 1. 计算缩放点积分数 (B, H, S_q, D) @ (B, H, D, S_k) -> (B, H, S_q, S_k)
    scores = np.matmul(q, k.swapaxes(-1, -2)) / np.sqrt(d_k)
    
    # 2. 注入因果掩码 (若需要)
    if is_causal:
        seq_len_q = q.shape[-2]
        seq_len_k = k.shape[-2]
        # 上三角为 True (未来信息需要被屏蔽)
        mask = np.triu(np.ones((seq_len_q, seq_len_k), dtype=bool), k=1)
        # 将被遮蔽区域赋为负无穷大
        scores = np.where(mask, -1e9, scores)
        
    # 3. 数值稳定 Softmax (减 max 归一化)
    scores_max = np.max(scores, axis=-1, keepdims=True)
    exp_scores = np.exp(scores - scores_max)
    attn_weights = exp_scores / np.sum(exp_scores, axis=-1, keepdims=True)
    
    # 4. 加权求和 (B, H, S_q, S_k) @ (B, H, S_k, D) -> (B, H, S_q, D)
    return np.matmul(attn_weights, v)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
B, H, S, D = 1, 1, 3, 4
q = np.random.randn(B, H, S, D)
k = np.random.randn(B, H, S, D)
v = np.random.randn(B, H, S, D)
out = scaled_dot_product_attention(q, k, v, is_causal=True)
assert out.shape == (B, H, S, D)
# 验证因果性:第 0 个输出只能看到第 0 个输入,改变第 1、2 个输入不影响 out[:, :, 0, :]
k_mod = k.copy()
k_mod[:, :, 1:, :] += 10.0
out_mod = scaled_dot_product_attention(q, k_mod, v, is_causal=True)
assert np.allclose(out[:, :, 0, :], out_mod[:, :, 0, :], atol=1e-5), "违反因果掩码隔离!"
print("✓ 缩放点积因果注意力自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:In FP16 mixed-precision training, why should you avoid masking with -1e30 or float('-inf')?
FP16 max finite magnitude is 65504, so -1e30 overflows to -inf. If an entire row is masked with -inf (e.g., full padding rows), (-inf) - (-inf) in Softmax produces NaN. Standard kernels use -1e4 or torch.finfo(dtype).min.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: NTK-Aware Scaled RoPE ExtrapolationBack to Part D & All 105Next: Multi-Head Attention (MHA) →