M5-007M5: NLP & Large Language ModelsPretraining Objectives & Data CurationEasy
⚡Flashcard Drill➔Mastery:
Causal Language Modeling (CLM) vs. Masked Language Modeling (MLM)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
CLM trains autoregressive decoders to predict the next token via unidirectional causal masking for text generation, whereas MLM trains bidirectional encoders to predict randomly masked tokens for discriminative representation learning.
📌 Key Takeaways
- •CLM (Causal LM / GPT / LLaMA): objective ; causal lower-triangular attention mask; natural alignment with autoregressive sequence generation
- •MLM (Masked LM / BERT / RoBERTa): objective ; bidirectional full attention; superior for discriminative representation and classification
- •Why CLM won the LLM revolution: CLM models unify generation, in-context few-shot learning, and reasoning into a single scalable next-token prediction objective
📐 Mathematical Derivations
Mathematical Mechanism: 1. Causal Language Modeling (Autoregressive): Maximum likelihood over sequence : Implemented via lower-triangular causal attention mask for . Every token position contributes a supervised training signal, achieving token supervision efficiency per forward pass. 2. Masked Language Modeling (Bidirectional): A random subset of tokens (typically ) is replaced with
[MASK] (80%), random token (10%), or unchanged (10%): The attention mask is fully bidirectional (). Only of tokens contribute to the loss function, requiring more tokens to achieve equivalent gradient updates compared to CLM.🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Supervision Efficiency: CLM trains on every single token ( token efficiency); MLM trains on only of tokens per sequence. MLM requires significantly more pre-training tokens to achieve the same total gradient update volume. ② In-Context Learning (ICL) Emergence: Autoregressive CLM models naturally learn to continue prompts conditionally, enabling zero-shot and few-shot in-context learning (). MLM models lack autoregressive factorization, rendering them incapable of natural conversational generation or multi-step reasoning. ③ Prefix-LM / Encoder-Decoder Middle Ground: Models like T5 or GLM use prefix-LM (bidirectional attention on the prompt, causal attention on the completion). While strong at summarization, decoder-only CLM scales more cleanly in multi-turn chat and KV-cache serving. ④ Pre-training vs Downstream Disparity in MLM: The
[MASK] token never appears in downstream real-world text, introducing a fundamental pre-training/inference distribution mismatch (partially mitigated by the 80/10/10 rule). ⑤ Interview Strategy: Contrast the loss objectives, compare token supervision efficiency ( vs ), and explain why CLM's autoregressive factorization is the foundational prerequisite for in-context learning and reasoning.⚠️ Common Interview Pitfalls
- ✕Assuming MLM is better for generation (bidirectional attention cannot generate coherent long text autoregressively without iterative non-causal Gibbs sampling)
- ✕Overlooking that MLM only computes loss on the masked tokens (much lower data efficiency per forward pass)
- ✕Failing to explain how the causal mask in CLM enables full parallel training of all positions simultaneously
🎯 Interviewer Follow-up Inquiries
- ?Why does the autoregressive factorization of CLM naturally yield in-context few-shot learning capabilities?
- ?What is Prefix-LM, and why did the industry converge on pure decoder-only CLM rather than Encoder-Decoder architectures?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚