← Back to LLMs Mind Map
⚡ LLMsID: flashattention

FlashAttention

🎯Core Definition
FlashAttention is an IO-aware exact attention implementation: it tiles QKTQ K^T into SRAM, uses online softmax with running max mim_i and normalization factor lil_i (recursively rescaled when merging blocks: mnew=max(m,mlocal)m_{new} = max(m, m_{local})), never writes the O(n2)O(n^2) attention matrix back to HBM in the forward pass, and recomputes rather than stores in the backward pass; the motivation is the ~10-20x bandwidth gap between HBM and SRAM (A100: 40GB HBM vs ~108KB SRAM).
💡Use Cases
the de-facto attention implementation for LLM training and inference (PyTorch SDPA, NVIDIA FA2/FA3, DeepSeek/Meta stacks); gains grow with sequence length, and long-context training essentially requires it.
⚡Key Problems Solved
standard implementations shuttle the attention matrix between HBM and SRAM repeatedly, making IO the bottleneck; tiling + recomputation cuts HBM traffic from O(n2d)O(n^2 d) to O(n2d2M)O(\frac{n^2 d^2}{M}) (MM = SRAM size), giving 2-4x speedup and O(n)O(n) memory instead of O(n2)O(n^2), with results bit-identical to standard softmax (not an approximation).
🎯Core Interview Exam Points
Fundamentals · Production Scenarios · Deep Dive
Core Def
Explain the core definition, underlying mechanism/math, and key principles of FlashAttention.
Use Cases
In production engineering, what are the primary use cases for FlashAttention and what core pain points does it solve?
🔥 High-Frequency Deep-Dive Questions
1
Why is attention memory-bound? The bandwidth/capacity gap between HBM and SRAM?
2
How does online softmax merge block statistics (recursive formulas for running max m and normalizer l)?
3
Why recompute in the backward pass instead of storing? Memory from O(n2)O(n^2) to what?
4
What did FlashAttention-2 improve over v1 (dropping intermediate matrices, fewer non-matmul ops, ~2x)?
5
Why is FlashAttention exact? Essential difference from sparse/low-rank attention?
Updated 2026-08-12
🎯
Verify Mastery: Practice Exam Questions for "FlashAttention"
Take pitfall multiple-choice questions with derivation options; mistakes are automatically logged.
🚀 Start Card Practice ➔
← Previous CardPre-LN & RMSNormNext Card →Supervised Fine-Tuning (SFT)

🔗 More LLMs Knowledge Cards

引用出处: TalentMe 450+ 工业级 AI 架构考点知识库 (Tech Vault) · 遵循 CC-BY-4.0 知识共享协议
URL: https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/flashattention
中文速览与面试要点

FlashAttention (FlashAttention)

定义: FlashAttention 是 IO 感知的精确注意力实现:把 Q K^T 分块 (tiling) 放进 SRAM 计算,配合 online softmax 维护运行最大值 m_i 与归一化因子 l_i(合并新块时递归缩放:m_new = max(m, m_local)),前向不把 O(n^2) 注意力矩阵写回 HBM,反向用重算而非存储;动机是 HBM 与 SRAM 的带宽差距约 10-20 倍(A100: 40GB HBM vs 约 108KB SRAM)。 使用场景: 大模型训练与推理的事实标准实现(PyTorch SDPA、NVIDIA FA2/FA3、DeepSeek/Meta 训练栈),序列越长收益越大,长上下文训练几乎必用。 解决的问题: 标准实现把注意力矩阵反复写读 HBM,IO 成为主要瓶颈;分块 + 重算把 HBM 访问从 O(n^2 d) 降到 O( n^2 d^2M)(M 为 SRAM 大小),速度提升 2-4 倍、显存从 O(n^2) 降到 O(n),且结果与标准 softmax 逐位一致(不是近似)。

核心要点

  • 为什么说 attention 是 memory-bound?HBM 与 SRAM 的带宽/容量差距?
  • online softmax 如何合并分块统计量?(运行最大值 m 与归一化因子 l 的递归公式)
  • 反向传播为什么重算而不存储?显存从 O(n^2) 降到多少?
  • FlashAttention-2 相比 v1 改进了什么?(省略中间矩阵、减少非矩阵运算、约 2×)?
  • FlashAttention 为什么是精确的?与稀疏/低秩注意力有什么本质区别?