M5-010M5: NLP & Large Language ModelsPretraining Objectives & Data CurationMedium
⚡Flashcard Drill➔Mastery:

Curriculum Learning and Data Ordering in LLM Pre-training

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-011 →
📐 Mathematical Definition
curriculum: Deasy→Dhard;or Dgeneral→Dspecialized\text{curriculum}:\ \mathcal{D}_{easy}\to\mathcal{D}_{hard};\qquad \text{or}\ \mathcal{D}_{general}\to\mathcal{D}_{specialized}
⚡ One-Sentence Core Answer
Curriculum learning structures pre-training by ordering data from broad, easy-to-learn web text to dense, high-quality code and academic reasoning, stabilizing early optimization and accelerating convergence.

📌 Key Takeaways

  • •
    Traditional curriculum vs LLM pre-training: pure sample-level difficulty sorting can cause catastrophic forgetting of earlier distributions; LLM curricula operate at the stage and domain mixture level
  • •
    Two-stage pre-training paradigm: Stage 1 trains on vast, diverse web text for broad linguistic foundations; Stage 2 ramps up high-quality synthetic, mathematical, and coding data
  • •
    Annealing phase: the final 5-10% of pre-training applies learning rate decay on ultra-curated reasoning data, producing dramatic benchmark performance jumps

📐 Mathematical Derivations

Mathematical Mechanism: 1. Dynamic Domain Weighting: Let the training budget be TT total tokens. At step t∈[0,T]t \in [0, T], domain mixture weights αk(t)\alpha_k(t) vary smoothly over time: Lt(θ)=∑k=1Kαk(t)Ex∼Dk[L(x;θ)],∑k=1Kαk(t)=1\mathcal{L}_t(\theta) = \sum_{k=1}^K \alpha_k(t) \mathbb{E}_{x \sim \mathcal{D}_k} \left[ \mathcal{L}(x; \theta) \right], \quad \sum_{k=1}^K \alpha_k(t) = 1 - Early Stage (t<0.7Tt < 0.7 T): αweb(t)≈0.7\alpha_{\text{web}}(t) \approx 0.7, αmath(t)≈0.05\alpha_{\text{math}}(t) \approx 0.05. - Late / Annealing Stage (t>0.9Tt > 0.9 T): αweb(t)≈0.2\alpha_{\text{web}}(t) \approx 0.2, αmath(t)≈0.35\alpha_{\text{math}}(t) \approx 0.35, αsynthetic(t)≈0.25\alpha_{\text{synthetic}}(t) \approx 0.25. 2. Learning Dynamics Rationale: In early training, the network prioritizes learning basic grammar, vocabulary frequencies, and short-range syntax (steepest gradient descent direction). Feeding dense mathematical proofs early can cause gradient instability and slow optimization. Once the base language representation is established, high-quality reasoning data shapes attention circuits efficiently.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Risk of Hard Stage Transitions: Abruptly switching datasets from web text to pure math causes catastrophic forgetting of general world knowledge. Domain ratios should transition smoothly via continuous interpolation curves rather than discrete step changes. ② Sequence Length Curriculum: Starting pre-training at shorter context lengths (e.g., 2048 or 4096 tokens) and ramping up to 8192+ in later stages saves massive attention compute during early iterations while preserving final long-context capability. ③ Final Annealing Impact (MiniCPM / LLaMA-3): Cooling down the learning rate to near-zero while feeding ultra-high-quality data produces a 'hockey stick' performance jump on MMLU, GSM8K, and HumanEval within just 50B-100B tokens. ④ Loss-Based Sample Filtering: Dynamic data pruning algorithms monitor individual sample loss during training; samples with near-zero loss (already mastered) or intractable noise (irreducible loss) are down-sampled. ⑤ Interview Strategy: Contrast naive sample-difficulty sorting with domain-mixture staging, explain the mathematical annealing schedule, and cite the hockey-stick benchmark uplift during the cooling phase.
⚠️ Common Interview Pitfalls
  • ✕
    Hard-switching datasets without a replay buffer or smooth interpolation (causes immediate catastrophic forgetting of early domains)
  • ✕
    Sorting individual sentences strictly by length or perplexity, which destroys document-level coherence and context
  • ✕
    Skipping the high-quality annealing phase at the end of the cosine learning rate decay
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the annealing phase at the tail of cosine learning rate decay produce outsized benchmark gains?
  • ?
    What criteria distinguish beneficial curriculum learning from harmful catastrophic forgetting in LLMs?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-009: Data Deduplication Methods and Why It Matters in LLM Pre-training📋 Back to BankNext →M5-011: Synthetic Data in Pre-training and Post-Training: Roles and Risks