M5-019M5: NLP & Large Language ModelsScaling Laws & Compute AllocationHard
⚡Flashcard Drill➔Mastery:
Post-Training Scaling (RL and Test-Time Compute) vs. Pre-Training Scaling
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Pre-training scaling expands general world knowledge via next-token prediction over static text, whereas post-training scaling expands complex reasoning through reinforcement learning exploration and test-time search computation.
📌 Key Takeaways
- •Pre-training scaling (System 1): predictable power laws over compute, parameters, and tokens; builds broad world knowledge, lexical fluency, and pattern recognition
- •Post-training RL scaling (System 2): reinforcement learning with verifiable rewards (math, code) enables self-discovery of extended Chain-of-Thought, error correction, and backtracking (OpenAI o1, DeepSeek-R1)
- •Test-Time compute scaling: spending additional inference compute per query (rejection sampling, best-of-, Monte Carlo tree search) trades inference latency for mathematical accuracy
📐 Mathematical Derivations
Mathematical Mechanism: 1. Pre-training Power Law: Loss follows passive unsupervised next-token prediction: Diminishing returns require more compute for incremental loss reductions. 2. Test-Time Compute Scaling Formulations: For complex problem , spending additional inference compute scales accuracy along two complementary axes: - Parallel Sampling (Best-of-$N$ with Verifier): Sample independent solutions and select the highest-scoring candidate via verifier : As expands, accuracy increases logarithmically with compute . - Sequential Thinking (Extended CoT / DeepSeek-R1): The model generates an internal monologue of reasoning tokens . Compute scales as . RL rewards longer reasoning trajectories when they successfully resolve edge cases and backtrack from errors.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Shift in Compute Economics: Instead of spending \100\text{M}\ and invest remaining compute into extensive post-training RL and dynamic test-time search, achieving superior reasoning at lower overall cost. ② Verifiable vs Subjective Domains: Post-training RL scaling works exceptionally well in domains with verifiable ground truth (coding tests, Olympiad mathematics, formal logic). In open-ended creative writing, lack of precise verifiers leads to reward hacking and sycophancy. ③ Inference Latency SLA: Test-time compute scaling directly inflates user latency (generating 5,000 reasoning tokens takes 20-40 seconds). Systems must route queries dynamically: use fast direct generation for simple questions and engage test-time reasoning only for complex multi-step problems. ④ Self-Correction Emergence: DeepSeek-R1 showed that purely through RL reward on answer correctness, models organically discover cognitive behaviors like backtracking ('Wait, let me double check this equation...'), self-questioning, and alternative exploration. ⑤ Interview Strategy: Contrast System 1 (intuitive pre-training next-token prediction) vs System 2 (deliberate test-time search and reasoning), explain the Best-of- and extended CoT scaling mechanics, and cite OpenAI o1 / DeepSeek-R1.
⚠️ Common Interview Pitfalls
- ✕Assuming post-training RL can replace pre-training (RL refines reasoning and search, but cannot inject factual world knowledge missing from pre-training)
- ✕Applying test-time search to open-ended creative tasks without objective outcome verifiers
- ✕Ignoring the user-facing latency and financial cost of generating thousands of internal thinking tokens per query
🎯 Interviewer Follow-up Inquiries
- ?How does DeepSeek-R1 incentivize the spontaneous emergence of self-correction behaviors without supervised human demonstrations?
- ?What is the mathematical trade-off between parallel Best-of- sampling and sequential extended reasoning tokens?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚