🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 101
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsM3: Deep Learning Foundations
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭Deep Learning Architecture Mindmap→📖Optimizers (AdamW/Lion) & Initialization→🎯Model Debugging→
Found 24 in-depth questions
M3-073Deep Learning FoundationsTraining Stability & Mixed Precision
Medium

Systematic Troubleshooting Workflow When Training Loss Becomes NaN

⚡ One-Sentence Answer

Isolate the first NaN occurrence via anomaly detection hooks; audit data inputs for corrupted values, inspect operators for division-by-zero or unmasked logits, and check FP16 overflow.

Math & Tradeoffs→
M3-074Deep Learning FoundationsTraining Stability & Mixed Precision
Hard

Operators Most Prone to Instability Under Low-Precision (FP16/BF16)

⚡ One-Sentence Answer

Operators involving exponentials (Softmax, GELU), logarithms (Cross-Entropy), division (LayerNorm), and massive summations (large-dim matrix multiplications) are most vulnerable.

Math & Tradeoffs→
M3-075Deep Learning FoundationsTraining Stability & Mixed Precision
Hard

Gradient Checkpointing (Activation Rematerialization): Memory vs Compute Trade-offs

⚡ One-Sentence Answer

It discards intermediate forward activations and recomputes them on-the-fly during the backward pass, trading $\sim 33\%$ extra compute to reduce activation memory from $O(L)$ to $O(\sqrt{L})$.

Math & Tradeoffs→
M3-076Deep Learning FoundationsTraining Stability & Mixed Precision
Medium

Attention Numerical Stability and How FlashAttention Solves It

⚡ One-Sentence Answer

Attention logits grow with sequence length and query-key norm, causing softmax overflow; FlashAttention uses online softmax tiling in SRAM to compute exact attention without materializing $O(N^2)$ matrices.

Math & Tradeoffs→
M3-077Deep Learning FoundationsTraining Stability & Mixed Precision
Medium

Why LayerNorm and Softmax Must Retain FP32 Precision in Mixed-Precision Training

⚡ One-Sentence Answer

LayerNorm and Softmax involve reductions and transcendental exponentials that trigger catastrophic cancellation and overflow in FP16; executing them in FP32 preserves numeric stability.

Math & Tradeoffs→
M3-078Deep Learning FoundationsDistributed Training Basics
Easy

Comparing Distributed Parallelism: Data (DP), Tensor (TP), and Pipeline (PP)

⚡ One-Sentence Answer

Data Parallelism splits batches across workers (All-Reduce); Tensor Parallelism shards weight matrices intra-node (All-Gather/Reduce-Scatter); Pipeline Parallelism partitions layers sequentially inter-node (P2P).

Math & Tradeoffs→
M3-079Deep Learning FoundationsDistributed Training Basics
Easy

ZeRO (Zero Redundancy Optimizer): The Three Sharding Stages Explained

⚡ One-Sentence Answer

ZeRO eliminates memory redundancies in distributed training by sharding optimizer states (ZeRO-1), gradients (ZeRO-2), and model parameters (ZeRO-3) across data-parallel ranks.

Math & Tradeoffs→
M3-080Deep Learning FoundationsDistributed Training Basics
Medium

Collective Communication Primitives: All-Reduce, All-Gather, and Reduce-Scatter

⚡ One-Sentence Answer

All-Reduce computes sum across all ranks and distributes the full result to all; Reduce-Scatter computes sum and scatters shards; All-Gather collects distributed shards onto every rank.

Math & Tradeoffs→
M3-081Deep Learning FoundationsDistributed Training Basics
Medium

Gradient Accumulation vs Data Parallelism: Mathematical Equivalence and Nuances

⚡ One-Sentence Answer

Both compute gradients across larger effective batches: Gradient Accumulation serializes micro-batches across time; Data Parallelism parallelizes micro-batches across space.

Math & Tradeoffs→
M3-082Deep Learning FoundationsDistributed Training Basics
Hard

Communication Bottlenecks in Large-Scale Training and Optimization Strategies

⚡ One-Sentence Answer

Communication bottlenecks arise from network bandwidth limits as models scale; mitigate via computation-communication overlap, gradient bucketing, FP8/FP16 compression, and topology-aware routing.

Math & Tradeoffs→
M3-083Deep Learning FoundationsDistributed Training Basics
Medium

FSDP vs DeepSpeed ZeRO: Architectural Relationship, Mechanics, and Differences

⚡ One-Sentence Answer

FSDP is PyTorch's native C++ implementation of ZeRO-3; both shard parameters, gradients, and optimizer states, but FSDP integrates directly with PyTorch autograd and Composable APIs.

Math & Tradeoffs→
M3-084Deep Learning FoundationsDistributed Training Basics
Hard

3D Parallelism and Sequence Parallelism (SP) in Large Foundation Models

⚡ One-Sentence Answer

3D Parallelism combines Data (DP), Tensor (TP), and Pipeline (PP) parallelism; Sequence Parallelism shards activation-heavy operations (LayerNorm, Dropout) along the sequence dimension.

Math & Tradeoffs→
M3-085Deep Learning FoundationsTraining Diagnostics & Debugging
Easy

Systematic Troubleshooting Workflow When Training Loss Fails to Decrease

⚡ One-Sentence Answer

First perform the single-batch overfitting sanity check; then systematically audit label correctness, learning rate schedules, initialization scale, gradient flow, and loss formulation.

Math & Tradeoffs→
M3-086Deep Learning FoundationsTraining Diagnostics & Debugging
Easy

Training Loss Decreases While Validation Loss Increases: Diagnosis and Remedies

⚡ One-Sentence Answer

This is textbook overfitting (high variance) or distribution drift; resolve by adding regularization, data augmentation, early stopping, or aligning domain shifts.

Math & Tradeoffs→
M3-087Deep Learning FoundationsTraining Diagnostics & Debugging
Medium

Three Common Loss Curve Anomalies and Their Underlying Root Causes

⚡ One-Sentence Answer

Loss curve anomalies include severe high-frequency oscillations (excessive LR), step-like plateau drops (scheduler/data pipeline shifts), and sudden vertical loss spikes (corrupted batches/numerical blowup).

Math & Tradeoffs→
M3-088Deep Learning FoundationsTraining Diagnostics & Debugging
Medium

How to Diagnose Underfitting vs Overfitting: Signals, Baselines, and Thresholds

⚡ One-Sentence Answer

Underfitting is diagnosed when training loss remains unacceptably high and close to validation loss; overfitting occurs when training loss is low while the generalization gap is wide.

Math & Tradeoffs→
M3-089Deep Learning FoundationsTraining Diagnostics & Debugging
Hard

Train-Serving Skew (Training-Inference Inconsistency): Root Causes and Mitigations

⚡ One-Sentence Answer

Train-serving skew occurs when models encounter divergent data distributions or execution behaviors between offline training and live production serving.

Math & Tradeoffs→
M3-090Deep Learning FoundationsTraining Diagnostics & Debugging
Medium

Statistical Diagnostics for Gradients and Activations: Distributions, Dead Units, and Logits

⚡ One-Sentence Answer

Inspect layer-wise activation means, standard deviations, dead neuron percentages, and logit dynamic ranges to detect vanishing, saturation, and representation collapse early.

Math & Tradeoffs→
M3-091Deep Learning FoundationsTraining Diagnostics & Debugging
Hard

Reproducibility Debugging in Deep Learning: RNG Seeds, Deterministic Algorithms, and Pipelines

⚡ One-Sentence Answer

Full reproducibility requires fixing random seeds across all libraries, enabling deterministic GPU algorithms, and isolating multi-processing data loader worker states.

Math & Tradeoffs→
M3-092Deep Learning FoundationsArchitecture Building Blocks
Easy

Why Residual Connections Enable Ultra-Deep Network Training

⚡ One-Sentence Answer

Residual connections break gradient vanishing by providing an additive identity pathway ($I$), creating an implicit ensemble of $2^L$ variable-depth paths and smoothing the loss landscape.

Math & Tradeoffs→
M3-093Deep Learning FoundationsArchitecture Building Blocks
Easy

The Bottleneck Architecture: Dimensionality Reduction, Expansion, and Efficiency

⚡ One-Sentence Answer

The bottleneck structure uses $1\times 1$ convolutions to compress channel dimensions before expensive $3\times 3$ spatial convolutions, slashing FLOPs and parameters while increasing non-linear depth.

Math & Tradeoffs→
M3-094Deep Learning FoundationsArchitecture Building Blocks
Medium

Depthwise Separable Convolutions: Mechanism and Computational Savings

⚡ One-Sentence Answer

It factorizes standard convolution into a channel-wise Depthwise convolution followed by a $1\times 1$ Pointwise convolution, reducing compute and parameters by factor $\frac{1}{C_{\text{out}}} + \frac{1}{K^2} \approx \frac{1}{K^2}$.

Math & Tradeoffs→
M3-095Deep Learning FoundationsArchitecture Building Blocks
Medium

Inductive Bias in Self-Attention vs Convolution

⚡ One-Sentence Answer

Convolution imposes rigid structural inductive biases (spatial locality, translation equivariance); Self-Attention is dynamic, content-based, and permutation-equivariant, trading prior structure for scaling capacity.

Math & Tradeoffs→
M3-096Deep Learning FoundationsArchitecture Building Blocks
Hard

Parameter Sharing in Deep Learning: Forms, Mechanisms, and Regularizing Impact

⚡ One-Sentence Answer

Parameter sharing constrains model capacity by reusing identical weights across space (CNNs), time (RNNs), layers (ALBERT), or modalities, injecting structural priors and slashing memory.

Math & Tradeoffs→