M4-008M4: Sequences & TransformersSeq2Seq & Attention OriginsEasy
⚡Flashcard Drill➔Mastery:
Bahdanau Additive Attention: Mathematical Workflow and Architectural Anatomy
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Bahdanau attention scores alignment via an additive MLP between decoder state and all encoder states, computing dynamic context vectors via softmax weighted averaging.
📌 Key Takeaways
- •Additive score formula:
- •Dynamic context vector: , where
- •Architectural distinction: uses decoder state (before emitting ) to query all bidirectional encoder states
📐 Mathematical Derivations
Algorithmic Workflow (Bahdanau, Cho, Bengio, ICLR 2015):
Let source annotations from a bidirectional RNN be for . Let previous decoder state be .
1. Alignment Energy Score (Additive MLP):
, where .
2. Attention Weight Distribution:
. Represents the probability that target token aligns to source token .
3. Context Vector Aggregation:
.
4. Decoder State & Output Generation:
Current decoder state: .
Output prediction: .
Let source annotations from a bidirectional RNN be for . Let previous decoder state be .
1. Alignment Energy Score (Additive MLP):
, where .
2. Attention Weight Distribution:
. Represents the probability that target token aligns to source token .
3. Context Vector Aggregation:
.
4. Decoder State & Output Generation:
Current decoder state: .
Output prediction: .
🏭 Production Trade-offs
Additive vs Dot-Product: Additive attention (Bahdanau) involves matrix multiplications followed by non-linear tanh. Multiplicative / Dot-Product attention (Luong, Vaswani) replaces the MLP with a single dot product , enabling hardware-accelerated BLAS matrix multiplications.
⚠️ Common Interview Pitfalls
- ✕Confusing Bahdanau additive attention with Luong multiplicative dot-product attention ()
- ✕Using current decoder state instead of previous state in Bahdanau attention calculations
🎯 Interviewer Follow-up Inquiries
- ?What computational advantages did Luong multiplicative dot-product attention offer over Bahdanau additive attention?
- ?Can attention weights be interpreted as rigorous causal alignments?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚