M4-008M4: Sequences & TransformersSeq2Seq & Attention OriginsEasy
⚡Flashcard Drill➔Mastery:

Bahdanau Additive Attention: Mathematical Workflow and Architectural Anatomy

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-009 →
📐 Mathematical Definition
et,i=v⊤tanh⁡(Wsst−1+Whhi);αt,i=eet,i∑jeet,j;ct=∑iαt,ihie_{t,i}=v^{\top}\tanh(W_s s_{t-1}+W_h h_i);\quad \alpha_{t,i}=\frac{e^{e_{t,i}}}{\sum_j e^{e_{t,j}}};\quad c_t=\sum_i\alpha_{t,i}h_i
⚡ One-Sentence Core Answer
Bahdanau attention scores alignment via an additive MLP between decoder state and all encoder states, computing dynamic context vectors via softmax weighted averaging.

📌 Key Takeaways

  • •
    Additive score formula: et,i=vaTtanh⁡(Wast−1+Uahi)e_{t, i} = v_a^T \tanh(W_a s_{t-1} + U_a h_i)
  • •
    Dynamic context vector: ct=∑i=1Txαt,ihic_t = \sum_{i=1}^{T_x} \alpha_{t, i} h_i, where αt,i=softmaxi(et,i)\alpha_{t, i} = \text{softmax}_i(e_{t, i})
  • •
    Architectural distinction: uses decoder state st−1s_{t-1} (before emitting yty_t) to query all bidirectional encoder states hih_i

📐 Mathematical Derivations

Algorithmic Workflow (Bahdanau, Cho, Bengio, ICLR 2015):
Let source annotations from a bidirectional RNN be hi=[h⃗i;h←i]h_i = [\vec{h}_i; \overleftarrow{h}_i] for i=1,…,Txi = 1, \dots, T_x. Let previous decoder state be st−1s_{t-1}.
1. Alignment Energy Score (Additive MLP):
et,i=vaTtanh⁡(Wast−1+Uahi+ba)e_{t, i} = v_a^T \tanh(W_a s_{t-1} + U_a h_i + b_a), where Wa∈Rda×ddec,Ua∈Rda×2denc,va∈RdaW_a \in \mathbb{R}^{d_a \times d_{\text{dec}}}, U_a \in \mathbb{R}^{d_a \times 2d_{\text{enc}}}, v_a \in \mathbb{R}^{d_a}.
2. Attention Weight Distribution:
αt,i=exp⁡(et,i)∑k=1Txexp⁡(et,k)\alpha_{t, i} = \frac{\exp(e_{t, i})}{\sum_{k=1}^{T_x} \exp(e_{t, k})}. Represents the probability that target token yty_t aligns to source token xix_i.
3. Context Vector Aggregation:
ct=∑i=1Txαt,ihic_t = \sum_{i=1}^{T_x} \alpha_{t, i} h_i.
4. Decoder State & Output Generation:
Current decoder state: st=f(st−1,[yt−1;ct])s_t = f(s_{t-1}, [y_{t-1}; c_t]).
Output prediction: P(yt∣y<t,X)=softmax(Wo[st;ct;yt−1])P(y_t \mid y_{<t}, X) = \text{softmax}\left( W_o [s_t; c_t; y_{t-1}] \right).

🏭 Production Trade-offs

Additive vs Dot-Product: Additive attention (Bahdanau) involves matrix multiplications followed by non-linear tanh. Multiplicative / Dot-Product attention (Luong, Vaswani) replaces the MLP with a single dot product sThs^T h, enabling hardware-accelerated BLAS matrix multiplications.
⚠️ Common Interview Pitfalls
  • ✕
    Confusing Bahdanau additive attention with Luong multiplicative dot-product attention (et,i=stTWahie_{t, i} = s_t^T W_a h_i)
  • ✕
    Using current decoder state sts_t instead of previous state st−1s_{t-1} in Bahdanau attention calculations
🎯 Interviewer Follow-up Inquiries
  • ?
    What computational advantages did Luong multiplicative dot-product attention offer over Bahdanau additive attention?
  • ?
    Can attention weights αt,i\alpha_{t, i} be interpreted as rigorous causal alignments?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-007: Seq2Seq Architecture and the Fixed-Vector Information Bottleneck📋 Back to BankNext →M4-009: Why Attention is Characterized as 'Differentiable Soft Alignment'