M5-028M5: NLP & Large Language ModelsInstruction Tuning & Supervised Fine-TuningHard
⚡Flashcard Drill➔Mastery:

Rejection Sampling Fine-Tuning (RFT / STaR) Mechanisms

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-029 →
📐 Mathematical Definition
Dnew={y∼πθ: verify(y)=1};θ←SFT(θ,Dnew)\mathcal{D}_{\text{new}}=\{y\sim\pi_\theta:\ \text{verify}(y)=1\};\qquad \theta\leftarrow\text{SFT}(\theta,\mathcal{D}_{\text{new}})
⚡ One-Sentence Core Answer
Rejection Sampling Fine-Tuning (RFT / STaR) samples multiple candidate reasoning paths from a model, filters them using automated verifiers, and fine-tunes the model on its own successful generations to bootstrap reasoning capabilities.

📌 Key Takeaways

  • •
    Core mechanism: for each problem xx, sample NN candidate solutions from policy πθ\pi_\theta; evaluate candidates with deterministic verifiers (unit tests, math ground truth); retain only correct paths
  • •
    Self-Taught Reasoner (STaR): pairs rejection sampling with rationalization (asking the model to generate a chain of thought when an answer is initially incorrect), creating a self-improving loop
  • •
    Bridge between SFT and RL: RFT acts as an offline, supervised surrogate for reinforcement learning, avoiding the training instability of PPO while harvesting diverse correct reasoning traces

📐 Mathematical Derivations

Mathematical Mechanism: 1. Rejection Sampling Generation Phase: For input problem xi∈Dx_i \in \mathcal{D} and current policy πθ\pi_\theta, sample KK candidate solutions with high temperature (T∈[0.7,1.0]T \in [0.7, 1.0]): yi,1,yi,2,…,yi,K∼πθ(⋅∣xi)y_{i, 1}, y_{i, 2}, \dots, y_{i, K} \sim \pi_\theta(\cdot \mid x_i) 2. Automated Verification Filtering: Evaluate candidate correctness using an oracle verifier V(x,y)∈{0,1}V(x, y) \in \{0, 1\} (e.g., Python execution or symbolic math check). Construct the filtered dataset D∗\mathcal{D}^*: D∗=⋃i{(xi,yi,k)∣V(xi,yi,k)=1}\mathcal{D}^* = \bigcup_{i} \left\{ (x_i, y_{i, k}) \mid V(x_i, y_{i, k}) = 1 \right\} 3. Fine-Tuning Update Phase: Fine-tune policy θ\theta on D∗\mathcal{D}^* via standard supervised cross-entropy: θ∗=arg⁡max⁡θ∑(x,y)∈D∗∑t=1∣y∣log⁡πθ(yt∣x,y<t)\theta^* = \arg\max_\theta \sum_{(x, y) \in \mathcal{D}^*} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{<t}) Iterating this process (t=1,2,…,Mt=1, 2, \dots, M) progressively bootstraps the model's reasoning accuracy.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Advantages over PPO / Online RL: RFT uses standard supervised learning pipelines (SFT), which are completely stable, highly parallelizable, and free from PPO hyperparameter volatility (value network training, clipping epsilon, advantage estimation). ② Solution Diversity vs Mode Collapse: High sampling temperature (T=0.8T=0.8) encourages the model to explore disparate reasoning strategies. Fine-tuning on diverse correct paths teaches the model multiple valid problem-solving routes. ③ The Exploration Frontier Limit: RFT cannot improve on problems where the model's pass@KK probability is exactly zero (P(success)=0P(\text{success}) = 0). To expand the frontier, problems must be paired with hints or seeded with teacher demonstrations. ④ Iterative Refinement (STaR / V-STaR): In each iteration, the updated model πθt+1\pi_{\theta_{t+1}} can solve harder problems that πθt\pi_{\theta_t} failed, expanding the training corpus D∗\mathcal{D}^* over successive rounds. ⑤ Interview Strategy: Detail the 3-step cycle (Sample →\to Verify →\to Fine-tune), contrast RFT with PPO, explain why deterministic verifiers are essential, and address the pass@K=0K=0 exploration bottleneck.
⚠️ Common Interview Pitfalls
  • ✕
    Using RFT in subjective domains without reliable ground-truth verifiers (leads to reinforcement of plausible hallucinations)
  • ✕
    Sampling with temperature T=0T=0 (yields KK identical outputs and zero exploration diversity)
  • ✕
    Ignoring the exploration limit when a model fails to generate even a single correct trace across all KK samples
🎯 Interviewer Follow-up Inquiries
  • ?
    How does STaR handle problems where the model fails to generate a correct solution during initial sampling?
  • ?
    Why is Rejection Sampling Fine-Tuning significantly more computationally stable than PPO in mathematical domains?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-027: Chat Template Formatting and Multi-Turn Conversation Construction📋 Back to BankNext →M5-029: SFT Data Sources and Engineering Trade-offs