🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 101
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsM3: Deep Learning Foundations
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭Deep Learning Architecture Mindmap→📖Optimizers (AdamW/Lion) & Initialization→🎯Model Debugging→
Found 24 in-depth questions
M3-049Deep Learning FoundationsRegularization & Training Tricks
Medium

Mixup and CutMix Data Augmentations: Mechanisms and Theoretical Foundations

⚡ One-Sentence Answer

Mixup blends pixel values and labels linearly; CutMix replaces bounding-box regions with image patches; both regularize models via Vicinal Risk Minimization.

Math & Tradeoffs→
M3-050Deep Learning FoundationsRegularization & Training Tricks
Medium

The Role of Exponential Moving Average (EMA) of Weights in Training

⚡ One-Sentence Answer

EMA tracks an exponential moving average of model parameters (shadow weights), smoothing stochastic optimization noise to deliver higher test stability and generalization.

Math & Tradeoffs→
M3-051Deep Learning FoundationsRegularization & Training Tricks
Hard

Stochastic Weight Averaging (SWA) and Weight Ensembling (Model Soups)

⚡ One-Sentence Answer

SWA averages checkpoint weights along a cyclical/constant learning rate trajectory to discover flatter minima; Model Soups blend weights of multiple fine-tuned models with zero extra inference cost.

Math & Tradeoffs→
M3-052Deep Learning FoundationsRegularization & Training Tricks
Medium

Label Smoothing: Role, Mathematical Mechanism, and Hidden Costs

⚡ One-Sentence Answer

Label smoothing softens one-hot targets to prevent logit explosion and model overconfidence, but impairs probability calibration and degrades teacher efficacy in knowledge distillation.

Math & Tradeoffs→
M3-053Deep Learning FoundationsRegularization & Training Tricks
Hard

Dropout Usage in Transformers and LLMs: Differences Across Modalities and Scales

⚡ One-Sentence Answer

Dropout prevents co-adaptation in small-data regimes, but is universally set to 0 in large language model pre-training to preserve model capacity and maximize token throughput.

Math & Tradeoffs→
M3-054Deep Learning FoundationsGradient Vanishing & Explosion
Easy

Vanishing and Exploding Gradients: Root Causes, Dynamics, and Modern Mitigations

⚡ One-Sentence Answer

Caused by deep Jacobian matrix chain multiplication where eigenvalues are consistently $<1$ or $>1$; resolved via residual connections, normalization, initialization, and gradient clipping.

Math & Tradeoffs→
M3-055Deep Learning FoundationsGradient Vanishing & Explosion
Easy

Gradient Clipping: Clip-by-Norm vs Clip-by-Value Differences and Geometry

⚡ One-Sentence Answer

Clip-by-value clamps individual elements into a bounding box, distorting gradient direction; clip-by-norm scales the entire gradient vector proportionally, strictly preserving directional trajectory.

Math & Tradeoffs→
M3-056Deep Learning FoundationsGradient Vanishing & Explosion
Medium

Gradient Noise Scale ($B^*$): Definition and Its Relationship to Optimal Batch Size

⚡ One-Sentence Answer

The Gradient Noise Scale ($B^*$) measures the ratio of gradient covariance trace to squared true gradient norm, identifying the critical batch size where data parallelism saturates.

Math & Tradeoffs→
M3-057Deep Learning FoundationsGradient Vanishing & Explosion
Medium

How to Detect and Isolate Gradient Anomalies: Diagnostic Signals and Workflows

⚡ One-Sentence Answer

Log layer-wise gradient norms, parameter norms, and update ratios using autograd hooks; diagnose vanishing (monotonic decay), explosion (spikes), and layer imbalances.

Math & Tradeoffs→
M3-058Deep Learning FoundationsGradient Vanishing & Explosion
Hard

Gradient Conflict in Multi-Task Learning and Remediation Techniques

⚡ One-Sentence Answer

Gradients from competing tasks point in opposing directions ($\langle g_i, g_j \rangle < 0$), causing destructive cancellation; mitigate via PCGrad orthogonal projection and GradNorm balance.

Math & Tradeoffs→
M3-059Deep Learning FoundationsGradient Vanishing & Explosion
Medium

How Residual Connections and Normalization Cooperate to Guarantee Gradient Flow

⚡ One-Sentence Answer

Residual connections provide an unbroken identity gradient highway ($\|J\| \sim O(1)$), while normalization stabilizes feature scales to keep sub-layer Jacobians well-conditioned.

Math & Tradeoffs→
M3-060Deep Learning FoundationsGradient Vanishing & Explosion
Hard

The Loss Spike Phenomenon: Gradient Norm Correlations, Root Causes, and Mitigations

⚡ One-Sentence Answer

Loss spikes are sudden 1-to-2 order-of-magnitude error explosions caused by corrupted data batches, outlier activation channels, or attention logit blowups; mitigate via checkpoint rollback and LR annealing.

Math & Tradeoffs→
M3-061Deep Learning FoundationsGradient Vanishing & Explosion
Hard

Gradient Scale Mismatch in Embedding and Output Layers: Logit Scaling and Tying

⚡ One-Sentence Answer

Output logits grow with hidden dimension $d$, saturating softmax gradients and creating extreme gradient scale discrepancies between embeddings and deep layers; resolve via logit scaling ($1/\sqrt{d}$) or soft-capping.

Math & Tradeoffs→
M3-062Deep Learning FoundationsGradient Vanishing & Explosion
Hard

Per-Layer and Adaptive Gradient Clipping (AGC) Explained

⚡ One-Sentence Answer

AGC clips gradients adaptively per layer based on the ratio of gradient norm to parameter norm, preventing layer-wise gradient explosions in Normalizer-Free networks (NFNet).

Math & Tradeoffs→
M3-063Deep Learning FoundationsGradient Vanishing & Explosion
Hard

Why Stochastic Gradient Noise Acts as Implicit Regularization in SGD

⚡ One-Sentence Answer

Stochastic mini-batch gradient noise acts as state-dependent Brownian motion that destabilizes sharp, narrow minima, driving parameters into broad, flat basins that generalize better.

Math & Tradeoffs→
M3-064Deep Learning FoundationsLoss Functions & Objectives
Easy

Comparing MSE, MAE, and Huber Loss: Robustness and Target Optimality

⚡ One-Sentence Answer

MSE optimizes for the conditional mean and penalizes outliers quadratically; MAE optimizes for the conditional median and is robust; Huber smoothly bridges both regimes.

Math & Tradeoffs→
M3-065Deep Learning FoundationsLoss Functions & Objectives
Easy

Cross-Entropy vs Binary Cross-Entropy (BCE) and Numerically Stable Implementations

⚡ One-Sentence Answer

BCE is the 2-class specialization of categorical cross-entropy; numeric stability requires fusing sigmoid/softmax with loss via the Log-Sum-Exp trick to prevent overflow.

Math & Tradeoffs→
M3-066Deep Learning FoundationsLoss Functions & Objectives
Medium

Focal Loss: Design Motivation, Mathematical Derivation, and Modulating Factor

⚡ One-Sentence Answer

Focal Loss adds a modulating factor $(1 - p_t)^\gamma$ to cross-entropy, dynamically down-weighting easy, well-classified examples to focus training on hard negative candidates.

Math & Tradeoffs→
M3-067Deep Learning FoundationsLoss Functions & Objectives
Medium

InfoNCE vs Triplet Loss in Contrastive Learning: Formulations and Information Capacity

⚡ One-Sentence Answer

Triplet Loss optimizes a single positive against a single negative via margin constraints; InfoNCE optimizes over multiple negatives simultaneously via temperature-scaled categorical cross-entropy.

Math & Tradeoffs→
M3-068Deep Learning FoundationsLoss Functions & Objectives
Hard

Loss Function and Evaluation Metric Mismatch: Causes, Examples, and Surrogates

⚡ One-Sentence Answer

Mismatch occurs when business metrics are non-differentiable (F1, AUC, NDCG) and models must train on smooth surrogate losses (CE, MSE) whose optimal points diverge.

Math & Tradeoffs→
M3-069Deep Learning FoundationsLoss Functions & Objectives
Medium

KL Divergence in Distillation and Regularization: The Asymmetry Problem

⚡ One-Sentence Answer

Forward KL $D_{\text{KL}}(P \parallel Q)$ enforces mode-covering (zero-avoiding); Reverse KL $D_{\text{KL}}(Q \parallel P)$ enforces mode-seeking (zero-forcing), critical in VAEs and RLHF.

Math & Tradeoffs→
M3-070Deep Learning FoundationsLoss Functions & Objectives
Hard

Ranking Losses in Search and Recommendation: Pointwise, Pairwise, and Listwise

⚡ One-Sentence Answer

Pointwise predicts independent scores; Pairwise optimizes relative order between document pairs; Listwise optimizes the entire ranked permutation directly.

Math & Tradeoffs→
M3-071Deep Learning FoundationsTraining Stability & Mixed Precision
Easy

Floating-Point Precisions in Deep Learning: FP16, BF16, and FP32 Trade-offs

⚡ One-Sentence Answer

FP32 offers 8-bit exponent and 23-bit mantissa; FP16 offers 5-bit exponent (prone to overflow) with 10-bit mantissa; BF16 matches FP32's 8-bit exponent (eliminating overflow) with 7-bit mantissa.

Math & Tradeoffs→
M3-072Deep Learning FoundationsTraining Stability & Mixed Precision
Medium

Loss Scaling Mechanism and Dynamic Adjustment in Mixed Precision

⚡ One-Sentence Answer

Loss scaling multiplies the forward loss by scale $S$ to push small backpropagated gradients into FP16 representable range, unscaling before the optimizer update.

Math & Tradeoffs→