M5-019M5: NLP & Large Language ModelsScaling Laws & Compute AllocationHard
⚡Flashcard Drill➔Mastery:

Post-Training Scaling (RL and Test-Time Compute) vs. Pre-Training Scaling

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-020 →
📐 Mathematical Definition
Lpretrain smooth power law;RL: task-dependent, less predictable\mathcal{L}_{\text{pretrain}}\ \text{smooth power law};\qquad \text{RL}:\ \text{task-dependent},\ \text{less predictable}
⚡ One-Sentence Core Answer
Pre-training scaling expands general world knowledge via next-token prediction over static text, whereas post-training scaling expands complex reasoning through reinforcement learning exploration and test-time search computation.

📌 Key Takeaways

  • •
    Pre-training scaling (System 1): predictable power laws over compute, parameters, and tokens; builds broad world knowledge, lexical fluency, and pattern recognition
  • •
    Post-training RL scaling (System 2): reinforcement learning with verifiable rewards (math, code) enables self-discovery of extended Chain-of-Thought, error correction, and backtracking (OpenAI o1, DeepSeek-R1)
  • •
    Test-Time compute scaling: spending additional inference compute per query (rejection sampling, best-of-NN, Monte Carlo tree search) trades inference latency for mathematical accuracy

📐 Mathematical Derivations

Mathematical Mechanism: 1. Pre-training Power Law: Loss follows passive unsupervised next-token prediction: L(Cpretrain)=E+(C0Cpretrain)α\mathcal{L}(C_{\text{pretrain}}) = E + \left(\frac{C_0}{C_{\text{pretrain}}}\right)^\alpha Diminishing returns require 10×10\times more compute for incremental loss reductions. 2. Test-Time Compute Scaling Formulations: For complex problem xx, spending additional inference compute CtestC_{\text{test}} scales accuracy along two complementary axes: - Parallel Sampling (Best-of-$N$ with Verifier): Sample NN independent solutions {y1,…,yN}∼Pθ(⋅∣x)\{y_1, \dots, y_N\} \sim P_\theta(\cdot \mid x) and select the highest-scoring candidate via verifier V(x,y)V(x, y): Pass@N=1−∏i=1N(1−P(yi is correct))\text{Pass@}N = 1 - \prod_{i=1}^N (1 - P(y_i \text{ is correct})) As NN expands, accuracy increases logarithmically with compute Ctest=N×CsingleC_{\text{test}} = N \times C_{\text{single}}. - Sequential Thinking (Extended CoT / DeepSeek-R1): The model generates an internal monologue of reasoning tokens TreasoningT_{\text{reasoning}}. Compute scales as Ctest=(Lprompt+Treasoning)×FLOPsC_{\text{test}} = (L_{\text{prompt}} + T_{\text{reasoning}}) \times \text{FLOPs}. RL rewards longer reasoning trajectories when they successfully resolve edge cases and backtrack from errors.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Shift in Compute Economics: Instead of spending \100\text{M}topre−traina500Bparametermodel,labscanpre−trainanefficient70Bmodelforto pre-train a 500B parameter model, labs can pre-train an efficient 70B model for\10M10\text{M} and invest remaining compute into extensive post-training RL and dynamic test-time search, achieving superior reasoning at lower overall cost. ② Verifiable vs Subjective Domains: Post-training RL scaling works exceptionally well in domains with verifiable ground truth (coding tests, Olympiad mathematics, formal logic). In open-ended creative writing, lack of precise verifiers leads to reward hacking and sycophancy. ③ Inference Latency SLA: Test-time compute scaling directly inflates user latency (generating 5,000 reasoning tokens takes 20-40 seconds). Systems must route queries dynamically: use fast direct generation for simple questions and engage test-time reasoning only for complex multi-step problems. ④ Self-Correction Emergence: DeepSeek-R1 showed that purely through RL reward on answer correctness, models organically discover cognitive behaviors like backtracking ('Wait, let me double check this equation...'), self-questioning, and alternative exploration. ⑤ Interview Strategy: Contrast System 1 (intuitive pre-training next-token prediction) vs System 2 (deliberate test-time search and reasoning), explain the Best-of-NN and extended CoT scaling mechanics, and cite OpenAI o1 / DeepSeek-R1.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming post-training RL can replace pre-training (RL refines reasoning and search, but cannot inject factual world knowledge missing from pre-training)
  • ✕
    Applying test-time search to open-ended creative tasks without objective outcome verifiers
  • ✕
    Ignoring the user-facing latency and financial cost of generating thousands of internal thinking tokens per query
🎯 Interviewer Follow-up Inquiries
  • ?
    How does DeepSeek-R1 incentivize the spontaneous emergence of self-correction behaviors without supervised human demonstrations?
  • ?
    What is the mathematical trade-off between parallel Best-of-NN sampling and sequential extended reasoning tokens?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-018: Guiding Hyperparameter and Architecture Choices via Scaling Laws📋 Back to BankNext →M5-020: Supervised Fine-Tuning (SFT) Objectives and Data Construction Principles