🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 136
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsM5: NLP & Large Language Models
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭LLM & Agent Pipelines Mindmap→📖RLHF, DPO & GRPO Reasoning→🎯AIE Fine-Tuning→
Found 24 in-depth questions
M5-001NLP & Large Language ModelsTokenization (BPE / WordPiece / Unigram)
Easy

BPE Algorithm and Why It Is Widely Used in LLMs

⚡ One-Sentence Answer

Byte-Pair Encoding (BPE) iteratively merges the most frequent pairs of adjacent characters or subwords to construct a fixed-size subword vocabulary, eliminating out-of-vocabulary words while balancing sequence length and vocabulary size.

Math & Tradeoffs→
M5-002NLP & Large Language ModelsTokenization (BPE / WordPiece / Unigram)
Easy

Comparison of BPE, WordPiece, and Unigram Tokenization

⚡ One-Sentence Answer

BPE merges subwords based on raw co-occurrence frequency, WordPiece merges subwords based on mutual information (maximum likelihood), and Unigram prunes a large initial candidate pool based on sentence-level likelihood loss.

Math & Tradeoffs→
M5-003NLP & Large Language ModelsTokenization (BPE / WordPiece / Unigram)
Medium

Vocabulary Size Trade-offs in LLMs

⚡ One-Sentence Answer

Larger vocabularies compress text into fewer tokens to accelerate inference and expand the effective context window, but increase embedding parameter size and output softmax latency.

Math & Tradeoffs→
M5-004NLP & Large Language ModelsTokenization (BPE / WordPiece / Unigram)
Medium

Impact of Tokenizer on Downstream Metrics and Why Cross-Model Perplexity (PPL) Comparisons Are Unfair

⚡ One-Sentence Answer

Perplexity is normalized per token, making raw cross-model PPL comparisons invalid across different tokenizers because models with smaller vocabularies split text into more, easily predictable subwords that artificially deflate token-level loss.

Math & Tradeoffs→
M5-005NLP & Large Language ModelsTokenization (BPE / WordPiece / Unigram)
Hard

Tokenizer Challenges in Multilingual and Code Domains

⚡ One-Sentence Answer

Multilingual and code tokenization suffer from script fertility imbalances, high UTF-8 byte fragmentation, and loss of indentation and whitespace structure, requiring byte-fallback and dedicated syntactic pre-tokenization.

Math & Tradeoffs→
M5-006NLP & Large Language ModelsTokenization (BPE / WordPiece / Unigram)
Hard

Vocabulary Expansion Techniques and Considerations

⚡ One-Sentence Answer

Vocabulary expansion adds domain- or language-specific subwords to a pre-trained model by resizing the embedding and LM head matrices, requiring careful weight initialization and targeted continual pre-training to prevent catastrophic forgetting.

Math & Tradeoffs→
M5-007NLP & Large Language ModelsPretraining Objectives & Data Curation
Easy

Causal Language Modeling (CLM) vs. Masked Language Modeling (MLM)

⚡ One-Sentence Answer

CLM trains autoregressive decoders to predict the next token via unidirectional causal masking for text generation, whereas MLM trains bidirectional encoders to predict randomly masked tokens for discriminative representation learning.

Math & Tradeoffs→
M5-008NLP & Large Language ModelsPretraining Objectives & Data Curation
Easy

Impact of Data Quality and Mixture Ratios on Pre-training

⚡ One-Sentence Answer

Pre-training quality is dominated by aggressive data filtering, semantic deduplication, and deliberate domain mixture ratios, where small amounts of high-quality synthetic, mathematical, and code data disproportionately drive reasoning capabilities.

Math & Tradeoffs→
M5-009NLP & Large Language ModelsPretraining Objectives & Data Curation
Medium

Data Deduplication Methods and Why It Matters in LLM Pre-training

⚡ One-Sentence Answer

Data deduplication removes verbatim and near-duplicate text via exact hash matching and MinHash LSH, preventing model memorization, privacy leakage, and inefficient compute waste.

Math & Tradeoffs→
M5-010NLP & Large Language ModelsPretraining Objectives & Data Curation
Medium

Curriculum Learning and Data Ordering in LLM Pre-training

⚡ One-Sentence Answer

Curriculum learning structures pre-training by ordering data from broad, easy-to-learn web text to dense, high-quality code and academic reasoning, stabilizing early optimization and accelerating convergence.

Math & Tradeoffs→
M5-011NLP & Large Language ModelsPretraining Objectives & Data Curation
Hard

Synthetic Data in Pre-training and Post-Training: Roles and Risks

⚡ One-Sentence Answer

Synthetic data expands training volume in data-scarce domains like math and code via model generation and automated verification, but carries severe risks of model collapse, diversity loss, and error amplification if unvalidated.

Math & Tradeoffs→
M5-012NLP & Large Language ModelsPretraining Objectives & Data Curation
Hard

Data Contamination and Its Impact on Benchmark Reliability

⚡ One-Sentence Answer

Data contamination occurs when benchmark test sets leak into pre-training or fine-tuning corpora, inflating evaluation scores through verbatim memorization and compromising the scientific validity of model comparisons.

Math & Tradeoffs→
M5-013NLP & Large Language ModelsPretraining Objectives & Data Curation
Medium

Pre-training Domain Mixtures and the Role of the Annealing Phase

⚡ One-Sentence Answer

The pre-training annealing phase decays the learning rate to near-zero while shifting the data mixture toward ultra-high-quality math, code, and curated reasoning corpora, achieving rapid benchmark uplift in the final 5% of training.

Math & Tradeoffs→
M5-014NLP & Large Language ModelsScaling Laws & Compute Allocation
Easy

Core Conclusions of Kaplan vs. Chinchilla Scaling Laws

⚡ One-Sentence Answer

Kaplan et al. concluded that model parameters should scale faster than training tokens ($N \propto C^{0.73}, D \propto C^{0.27}$), while Chinchilla proved that parameters and tokens should scale in equal proportion ($N \propto C^{0.5}, D \propto C^{0.5}$, $\approx 20$ tokens per parameter).

Math & Tradeoffs→
M5-015NLP & Large Language ModelsScaling Laws & Compute Allocation
Easy

Why Inference Costs Alter Compute-Optimal Scaling Choices

⚡ One-Sentence Answer

Chinchilla optimality minimizes training compute alone, but production systems minimize total lifecycle cost (training + inference), justifying massive over-training of compact models to achieve low serving latency and high throughput.

Math & Tradeoffs→
M5-016NLP & Large Language ModelsScaling Laws & Compute Allocation
Medium

Emergent Abilities in Large Language Models and the Metric Discontinuity Debate

⚡ One-Sentence Answer

Emergent abilities describe capabilities that appear abruptly at scale, but Schaeffer et al. revealed that this apparent sharp discontinuity is largely an artifact of non-linear step-function evaluation metrics rather than a discontinuous shift in underlying model representations.

Math & Tradeoffs→
M5-017NLP & Large Language ModelsScaling Laws & Compute Allocation
Medium

Scaling Strategies Under Data-Constrained Regimes

⚡ One-Sentence Answer

As high-quality public human text approaches exhaustion, scaling strategies shift toward multi-epoch pre-training, aggressive synthetic data synthesis, multimodal data expansion, and test-time reasoning compute.

Math & Tradeoffs→
M5-018NLP & Large Language ModelsScaling Laws & Compute Allocation
Hard

Guiding Hyperparameter and Architecture Choices via Scaling Laws

⚡ One-Sentence Answer

Scaling laws guide pre-training hyperparameter tuning by using Maximal Update Parametrization ($\mu$P) to transfer optimal learning rates from tiny proxy models directly to hundred-billion-parameter models without expensive hyperparameter sweeps.

Math & Tradeoffs→
M5-019NLP & Large Language ModelsScaling Laws & Compute Allocation
Hard

Post-Training Scaling (RL and Test-Time Compute) vs. Pre-Training Scaling

⚡ One-Sentence Answer

Pre-training scaling expands general world knowledge via next-token prediction over static text, whereas post-training scaling expands complex reasoning through reinforcement learning exploration and test-time search computation.

Math & Tradeoffs→
M5-020NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Easy

Supervised Fine-Tuning (SFT) Objectives and Data Construction Principles

⚡ One-Sentence Answer

SFT adapts pre-trained base models into conversational assistants by optimizing next-token cross-entropy exclusively over target assistant responses, requiring high instruction diversity, clear formatting, and strict factual quality.

Math & Tradeoffs→
M5-021NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Easy

Loss Masking and Sequence Packing in SFT Training

⚡ One-Sentence Answer

Loss masking restricts gradient backpropagation exclusively to response tokens, while sequence packing concatenates multiple conversations into a single sequence with block-diagonal attention masks to eliminate padding waste.

Math & Tradeoffs→
M5-022NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Medium

Catastrophic Forgetting in SFT and Mitigation Strategies

⚡ One-Sentence Answer

Catastrophic forgetting occurs when fine-tuning on narrow instruction datasets overwrites pre-trained general knowledge and reasoning circuits, mitigated through pre-training data replay, low learning rates, and parameter-efficient tuning.

Math & Tradeoffs→
M5-023NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Medium

Chain-of-Thought (CoT) Distillation: Benefits and Hallucination Risks

⚡ One-Sentence Answer

CoT distillation transfers multi-step reasoning capabilities from teacher models to compact students by training on intermediate reasoning steps, but risks superficial format mimicry and unverified hallucinations if reasoning tokens are not anchored in verification.

Math & Tradeoffs→
M5-024NLP & Large Language ModelsInstruction Tuning & Supervised Fine-Tuning
Hard

SFT Data Mixture and Multi-Task Balancing

⚡ One-Sentence Answer

SFT multi-task balancing allocates data proportions across dialogue, coding, mathematics, reasoning, and safety to prevent over-fitting to single modalities while maintaining generalist conversational capabilities.

Math & Tradeoffs→