🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 938
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsGlobal System
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭Math Foundations Mindmap & Cards→📖Bayesian Inference, Entropy & KL Divergence→🎯DS Track→
Found 24 in-depth questions
M3-057Deep Learning FoundationsGradient Vanishing & Explosion
Medium

How to Detect and Isolate Gradient Anomalies: Diagnostic Signals and Workflows

⚡ One-Sentence Answer

Log layer-wise gradient norms, parameter norms, and update ratios using autograd hooks; diagnose vanishing (monotonic decay), explosion (spikes), and layer imbalances.

Math & Tradeoffs→
M3-058Deep Learning FoundationsGradient Vanishing & Explosion
Hard

Gradient Conflict in Multi-Task Learning and Remediation Techniques

⚡ One-Sentence Answer

Gradients from competing tasks point in opposing directions ($\langle g_i, g_j \rangle < 0$), causing destructive cancellation; mitigate via PCGrad orthogonal projection and GradNorm balance.

Math & Tradeoffs→
M3-059Deep Learning FoundationsGradient Vanishing & Explosion
Medium

How Residual Connections and Normalization Cooperate to Guarantee Gradient Flow

⚡ One-Sentence Answer

Residual connections provide an unbroken identity gradient highway ($\|J\| \sim O(1)$), while normalization stabilizes feature scales to keep sub-layer Jacobians well-conditioned.

Math & Tradeoffs→
M3-060Deep Learning FoundationsGradient Vanishing & Explosion
Hard

The Loss Spike Phenomenon: Gradient Norm Correlations, Root Causes, and Mitigations

⚡ One-Sentence Answer

Loss spikes are sudden 1-to-2 order-of-magnitude error explosions caused by corrupted data batches, outlier activation channels, or attention logit blowups; mitigate via checkpoint rollback and LR annealing.

Math & Tradeoffs→
M3-061Deep Learning FoundationsGradient Vanishing & Explosion
Hard

Gradient Scale Mismatch in Embedding and Output Layers: Logit Scaling and Tying

⚡ One-Sentence Answer

Output logits grow with hidden dimension $d$, saturating softmax gradients and creating extreme gradient scale discrepancies between embeddings and deep layers; resolve via logit scaling ($1/\sqrt{d}$) or soft-capping.

Math & Tradeoffs→
M3-062Deep Learning FoundationsGradient Vanishing & Explosion
Hard

Per-Layer and Adaptive Gradient Clipping (AGC) Explained

⚡ One-Sentence Answer

AGC clips gradients adaptively per layer based on the ratio of gradient norm to parameter norm, preventing layer-wise gradient explosions in Normalizer-Free networks (NFNet).

Math & Tradeoffs→
M3-063Deep Learning FoundationsGradient Vanishing & Explosion
Hard

Why Stochastic Gradient Noise Acts as Implicit Regularization in SGD

⚡ One-Sentence Answer

Stochastic mini-batch gradient noise acts as state-dependent Brownian motion that destabilizes sharp, narrow minima, driving parameters into broad, flat basins that generalize better.

Math & Tradeoffs→
M3-064Deep Learning FoundationsLoss Functions & Objectives
Easy

Comparing MSE, MAE, and Huber Loss: Robustness and Target Optimality

⚡ One-Sentence Answer

MSE optimizes for the conditional mean and penalizes outliers quadratically; MAE optimizes for the conditional median and is robust; Huber smoothly bridges both regimes.

Math & Tradeoffs→
M3-065Deep Learning FoundationsLoss Functions & Objectives
Easy

Cross-Entropy vs Binary Cross-Entropy (BCE) and Numerically Stable Implementations

⚡ One-Sentence Answer

BCE is the 2-class specialization of categorical cross-entropy; numeric stability requires fusing sigmoid/softmax with loss via the Log-Sum-Exp trick to prevent overflow.

Math & Tradeoffs→
M3-066Deep Learning FoundationsLoss Functions & Objectives
Medium

Focal Loss: Design Motivation, Mathematical Derivation, and Modulating Factor

⚡ One-Sentence Answer

Focal Loss adds a modulating factor $(1 - p_t)^\gamma$ to cross-entropy, dynamically down-weighting easy, well-classified examples to focus training on hard negative candidates.

Math & Tradeoffs→
M3-067Deep Learning FoundationsLoss Functions & Objectives
Medium

InfoNCE vs Triplet Loss in Contrastive Learning: Formulations and Information Capacity

⚡ One-Sentence Answer

Triplet Loss optimizes a single positive against a single negative via margin constraints; InfoNCE optimizes over multiple negatives simultaneously via temperature-scaled categorical cross-entropy.

Math & Tradeoffs→
M3-068Deep Learning FoundationsLoss Functions & Objectives
Hard

Loss Function and Evaluation Metric Mismatch: Causes, Examples, and Surrogates

⚡ One-Sentence Answer

Mismatch occurs when business metrics are non-differentiable (F1, AUC, NDCG) and models must train on smooth surrogate losses (CE, MSE) whose optimal points diverge.

Math & Tradeoffs→
M3-069Deep Learning FoundationsLoss Functions & Objectives
Medium

KL Divergence in Distillation and Regularization: The Asymmetry Problem

⚡ One-Sentence Answer

Forward KL $D_{\text{KL}}(P \parallel Q)$ enforces mode-covering (zero-avoiding); Reverse KL $D_{\text{KL}}(Q \parallel P)$ enforces mode-seeking (zero-forcing), critical in VAEs and RLHF.

Math & Tradeoffs→
M3-070Deep Learning FoundationsLoss Functions & Objectives
Hard

Ranking Losses in Search and Recommendation: Pointwise, Pairwise, and Listwise

⚡ One-Sentence Answer

Pointwise predicts independent scores; Pairwise optimizes relative order between document pairs; Listwise optimizes the entire ranked permutation directly.

Math & Tradeoffs→
M3-071Deep Learning FoundationsTraining Stability & Mixed Precision
Easy

Floating-Point Precisions in Deep Learning: FP16, BF16, and FP32 Trade-offs

⚡ One-Sentence Answer

FP32 offers 8-bit exponent and 23-bit mantissa; FP16 offers 5-bit exponent (prone to overflow) with 10-bit mantissa; BF16 matches FP32's 8-bit exponent (eliminating overflow) with 7-bit mantissa.

Math & Tradeoffs→
M3-072Deep Learning FoundationsTraining Stability & Mixed Precision
Medium

Loss Scaling Mechanism and Dynamic Adjustment in Mixed Precision

⚡ One-Sentence Answer

Loss scaling multiplies the forward loss by scale $S$ to push small backpropagated gradients into FP16 representable range, unscaling before the optimizer update.

Math & Tradeoffs→
M3-073Deep Learning FoundationsTraining Stability & Mixed Precision
Medium

Systematic Troubleshooting Workflow When Training Loss Becomes NaN

⚡ One-Sentence Answer

Isolate the first NaN occurrence via anomaly detection hooks; audit data inputs for corrupted values, inspect operators for division-by-zero or unmasked logits, and check FP16 overflow.

Math & Tradeoffs→
M3-074Deep Learning FoundationsTraining Stability & Mixed Precision
Hard

Operators Most Prone to Instability Under Low-Precision (FP16/BF16)

⚡ One-Sentence Answer

Operators involving exponentials (Softmax, GELU), logarithms (Cross-Entropy), division (LayerNorm), and massive summations (large-dim matrix multiplications) are most vulnerable.

Math & Tradeoffs→
M3-075Deep Learning FoundationsTraining Stability & Mixed Precision
Hard

Gradient Checkpointing (Activation Rematerialization): Memory vs Compute Trade-offs

⚡ One-Sentence Answer

It discards intermediate forward activations and recomputes them on-the-fly during the backward pass, trading $\sim 33\%$ extra compute to reduce activation memory from $O(L)$ to $O(\sqrt{L})$.

Math & Tradeoffs→
M3-076Deep Learning FoundationsTraining Stability & Mixed Precision
Medium

Attention Numerical Stability and How FlashAttention Solves It

⚡ One-Sentence Answer

Attention logits grow with sequence length and query-key norm, causing softmax overflow; FlashAttention uses online softmax tiling in SRAM to compute exact attention without materializing $O(N^2)$ matrices.

Math & Tradeoffs→
M3-077Deep Learning FoundationsTraining Stability & Mixed Precision
Medium

Why LayerNorm and Softmax Must Retain FP32 Precision in Mixed-Precision Training

⚡ One-Sentence Answer

LayerNorm and Softmax involve reductions and transcendental exponentials that trigger catastrophic cancellation and overflow in FP16; executing them in FP32 preserves numeric stability.

Math & Tradeoffs→
M3-078Deep Learning FoundationsDistributed Training Basics
Easy

Comparing Distributed Parallelism: Data (DP), Tensor (TP), and Pipeline (PP)

⚡ One-Sentence Answer

Data Parallelism splits batches across workers (All-Reduce); Tensor Parallelism shards weight matrices intra-node (All-Gather/Reduce-Scatter); Pipeline Parallelism partitions layers sequentially inter-node (P2P).

Math & Tradeoffs→
M3-079Deep Learning FoundationsDistributed Training Basics
Easy

ZeRO (Zero Redundancy Optimizer): The Three Sharding Stages Explained

⚡ One-Sentence Answer

ZeRO eliminates memory redundancies in distributed training by sharding optimizer states (ZeRO-1), gradients (ZeRO-2), and model parameters (ZeRO-3) across data-parallel ranks.

Math & Tradeoffs→
M3-080Deep Learning FoundationsDistributed Training Basics
Medium

Collective Communication Primitives: All-Reduce, All-Gather, and Reduce-Scatter

⚡ One-Sentence Answer

All-Reduce computes sum across all ranks and distributes the full result to all; Reduce-Scatter computes sum and scatters shards; All-Gather collects distributed shards onto every rank.

Math & Tradeoffs→