🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 136
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsM5: NLP & Large Language Models
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭LLM & Agent Pipelines Mindmap→📖RLHF, DPO & GRPO Reasoning→🎯AIE Fine-Tuning→
Found 24 in-depth questions
M5-025NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Hard

Division of Labor Between SFT and RLHF/DPO

⚡ One-Sentence Answer

SFT establishes foundational instruction following, domain knowledge, and response formatting, whereas RLHF/DPO refines preference boundaries, suppresses low-probability tail errors, and aligns model outputs with human values.

Math & Tradeoffs→
M5-026NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Medium

Choosing Learning Rates and Epochs in SFT

⚡ One-Sentence Answer

Supervised fine-tuning requires conservative learning rates ($10\times$ to $50\times$ smaller than pre-training) and low epoch counts (1 to 3 epochs) with cosine decay to adapt conversational formatting without destroying pre-trained knowledge representations.

Math & Tradeoffs→
M5-027NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Medium

Chat Template Formatting and Multi-Turn Conversation Construction

⚡ One-Sentence Answer

Chat templates serialize multi-turn conversational roles into unambiguous token streams using special delimiter tokens and Jinja2 formatting, ensuring strict boundary separation between system, user, and assistant turns.

Math & Tradeoffs→
M5-028NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Hard

Rejection Sampling Fine-Tuning (RFT / STaR) Mechanisms

⚡ One-Sentence Answer

Rejection Sampling Fine-Tuning (RFT / STaR) samples multiple candidate reasoning paths from a model, filters them using automated verifiers, and fine-tunes the model on its own successful generations to bootstrap reasoning capabilities.

Math & Tradeoffs→
M5-029NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Medium

SFT Data Sources and Engineering Trade-offs

⚡ One-Sentence Answer

SFT datasets combine four complementary sources: human expert annotation for high-quality instruction adherence, strong model distillation for scalable volume, open-source benchmarks for baseline coverage, and verified synthetic generation for complex reasoning.

Math & Tradeoffs→
M5-030NLP & Large Language ModelsAlignment & RLHF
Easy

The Three-Stage RLHF Pipeline

⚡ One-Sentence Answer

The canonical RLHF pipeline consists of three sequential phases: Supervised Fine-Tuning to establish instruction following, Reward Modeling on human preference pairs, and PPO Reinforcement Learning with a KL penalty to optimize policy generation.

Math & Tradeoffs→
M5-031NLP & Large Language ModelsAlignment & RLHF
Easy

Bradley-Terry Preference Model and Reward Model Loss Formulation

⚡ One-Sentence Answer

The Bradley-Terry model parameterizes pairwise preference probability as the sigmoid of reward differences, training the reward model via binary cross-entropy loss where only relative reward margins are statistically identifiable.

Math & Tradeoffs→
M5-032NLP & Large Language ModelsAlignment & RLHF
Medium

PPO Clipped Surrogate Objective and the Rationale for Min and Clip

⚡ One-Sentence Answer

PPO limits policy updates by clipping the probability ratio to $[1-\epsilon, 1+\epsilon]$ and taking the minimum against the unclipped objective, forming a pessimistic lower bound that prevents destructive policy collapse.

Math & Tradeoffs→
M5-033NLP & Large Language ModelsAlignment & RLHF
Medium

Generalized Advantage Estimation (GAE) and Advantage in RLHF

⚡ One-Sentence Answer

GAE calculates the advantage function by computing an exponentially weighted average of multi-step temporal difference residuals, balancing bias and variance via hyperparameter $\lambda$ to guide policy updates.

Math & Tradeoffs→
M5-034NLP & Large Language ModelsAlignment & RLHF
Hard

Reward Hacking and Mitigation Strategies in RLHF

⚡ One-Sentence Answer

Reward hacking occurs when a policy exploits flaws in the proxy reward model to score high rewards without improving true response quality, mitigated through KL penalties, reward ensembles, length penalties, and iterative reward model updates.

Math & Tradeoffs→
M5-035NLP & Large Language ModelsAlignment & RLHF
Hard

Engineering Challenges and Stability Issues in RLHF

⚡ One-Sentence Answer

RLHF is notoriously complex to scale because it requires orchestrating four large models across distributed GPU clusters while managing high policy variance, value network divergence, and reward model exploitation.

Math & Tradeoffs→
M5-036NLP & Large Language ModelsAlignment & RLHF
Medium

The Role and Dynamics of the KL Penalty Coefficient in RLHF

⚡ One-Sentence Answer

The KL penalty coefficient $\beta$ bounds policy divergence from the reference model to prevent reward hacking and language degeneration, balancing exploration against safety constraints.

Math & Tradeoffs→
M5-037NLP & Large Language ModelsAlignment & RLHF
Hard

Reward Model Generalization and Over-Optimization

⚡ One-Sentence Answer

Aggressive policy optimization against a static reward model inevitably leads to over-optimization, where policy generations exploit reward model errors and out-of-distribution artifacts to achieve high scores despite declining human quality.

Math & Tradeoffs→
M5-038NLP & Large Language ModelsAlignment & RLHF
Hard

On-Policy Sampling vs. Off-Policy Data in RLHF

⚡ One-Sentence Answer

RLHF requires on-policy sampling so that policy updates correspond strictly to the current distribution of generated tokens, while off-policy buffer reuse introduces severe distribution shifts and advantage estimation bias.

Math & Tradeoffs→
M5-039NLP & Large Language ModelsAlignment & RLHF
Hard

PPO Implementation Nuances in LLMs: Token-Level vs. Sequence-Level

⚡ One-Sentence Answer

PPO in LLMs operates over a hybrid token-sequence hierarchy where rewards are evaluated at the sequence level, KL penalties are computed at the token level, and advantages are attributed per-token via the Critic value network.

Math & Tradeoffs→
M5-040NLP & Large Language ModelsAlignment & RLHF
Hard

Motivations and Practices of Iterative / Online RLHF

⚡ One-Sentence Answer

Iterative (Online) RLHF periodically retrains the reward model on fresh completions sampled from the current policy, preventing out-of-distribution reward exploitation and continually aligning the model across training rounds.

Math & Tradeoffs→
M5-041NLP & Large Language ModelsAlignment & RLHF
Medium

Constitutional AI and RLAIF (Reinforcement Learning from AI Feedback)

⚡ One-Sentence Answer

Constitutional AI aligns language models by replacing human annotators with an AI feedback loop governed by explicit constitutional principles, automating harmlessness critique, revision, and preference labeling at scale.

Math & Tradeoffs→
M5-042NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)
Easy

Full Mathematical Derivation of Direct Preference Optimization (DPO) Loss

⚡ One-Sentence Answer

DPO analytically solves the optimal policy under the KL-regularized RL objective, expressing the ground-truth reward function directly in terms of policy and reference log-probabilities to eliminate the reward model entirely.

Math & Tradeoffs→
M5-043NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)
Easy

Systematic Comparison of PPO, DPO, and GRPO

⚡ One-Sentence Answer

PPO uses an explicit Reward Model and Critic network for online policy gradients, DPO uses offline preference pairs to optimize an implicit reward without reinforcement learning, and GRPO eliminates the Critic by computing relative advantages across prompt rollout groups.

Math & Tradeoffs→
M5-044NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)
Medium

Motivations and Distinctions of ORPO and KTO

⚡ One-Sentence Answer

ORPO integrates preference alignment directly into SFT via an odds-ratio penalty without a reference model, while KTO optimizes directly on unpaired thumbs-up/thumbs-down binary feedback using Kahneman-Tversky prospect theory.

Math & Tradeoffs→
M5-045NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)
Medium

SimPO: Simple Preference Optimization Improvements

⚡ One-Sentence Answer

SimPO simplifies preference alignment by eliminating the reference model entirely and introducing a length-normalized implicit reward with a target margin $\gamma$, achieving superior alignment with lower memory and zero length exploitation.

Math & Tradeoffs→
M5-046NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)
Hard

Failure Modes and Fundamental Limitations of DPO

⚡ One-Sentence Answer

DPO's primary limitations stem from its offline nature, leading to out-of-distribution generation drift, vulnerability to preference label noise, severe verbosity exploitation, and likelihood displacement.

Math & Tradeoffs→
M5-047NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)
Hard

Preference Data Construction and Quality Control

⚡ One-Sentence Answer

High-quality preference data requires meaningful quality deltas between completions, strict length bias equalization, and robust noise filtering across annotators or automated LLM judges.

Math & Tradeoffs→
M5-048NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)
Medium

Identity Preference Optimization (IPO) Motivation and Regularization

⚡ One-Sentence Answer

IPO replaces DPO's unconstrained cross-entropy loss with a regularized mean-squared error objective, bounding the implicit reward gap to prevent policy over-fitting and likelihood displacement.

Math & Tradeoffs→