🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 938
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsGlobal System
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭Math Foundations Mindmap & Cards→📖Bayesian Inference, Entropy & KL Divergence→🎯DS Track→
Found 24 in-depth questions
M3-009Deep Learning FoundationsWeight Initialization
Easy

Xavier and Kaiming Initialization: Mathematical Variances and Suitable Activations

⚡ One-Sentence Answer

Xavier sets $\text{Var}(W) = \frac{2}{n_{\text{in}} + n_{\text{out}}}$ for linear/Tanh activations; Kaiming sets $\text{Var}(W) = \frac{2}{n_{\text{in}}}$ to compensate for ReLU's half-space zeroing.

Math & Tradeoffs→
M3-010Deep Learning FoundationsWeight Initialization
Medium

The 'Variance Preservation' Principle and Its Relation to Gradient Stability

⚡ One-Sentence Answer

Maintaining unit activation and gradient variance across successive layers prevents exponential signal explosion or vanishing in deep networks.

Math & Tradeoffs→
M3-011Deep Learning FoundationsWeight Initialization
Medium

Common Initialization Techniques in Transformers

⚡ One-Sentence Answer

Key techniques include residual branch scaling ($1/\sqrt{2L}$), zero-initializing output projections, embedding rescaling, and small initial LayerNorm gains.

Math & Tradeoffs→
M3-012Deep Learning FoundationsWeight Initialization
Hard

Maximal Update Parametrization (μP): Concepts and Hyperparameter Transfer

⚡ One-Sentence Answer

μP adjusts weight initialization and learning rates by layer width, ensuring optimal hyperparameters (learning rate, schedule) remain invariant as model width scales.

Math & Tradeoffs→
M3-013Deep Learning FoundationsActivation Functions
Easy

Comparing Sigmoid, Tanh, and ReLU: Mathematical Properties and Trade-offs

⚡ One-Sentence Answer

Sigmoid and Tanh suffer from gradient saturation and computational cost; ReLU solves vanishing gradients with fast piece-wise linearity but risks dying neurons.

Math & Tradeoffs→
M3-014Deep Learning FoundationsActivation Functions
Easy

The Dying ReLU Problem and How LeakyReLU, ELU, and GELU Mitigate It

⚡ One-Sentence Answer

Dying ReLU occurs when neurons receive negative inputs causing zero gradients and permanent deactivation; LeakyReLU, ELU, and GELU resolve this by ensuring non-zero gradients for negative inputs.

Math & Tradeoffs→
M3-015Deep Learning FoundationsActivation Functions
Medium

Why Modern Transformers Use GELU and SiLU Instead of ReLU

⚡ One-Sentence Answer

GELU and SiLU offer smooth non-monotonic curvature, continuous higher-order derivatives, and probabilistic gating interpretations that eliminate gradient shattering and dying neurons.

Math & Tradeoffs→
M3-016Deep Learning FoundationsActivation Functions
Medium

The Gated Linear Unit (GLU) Family and Why They Excel in Large Language Models

⚡ One-Sentence Answer

GLUs compute element-wise products between a linear projection and a non-linearly gated projection; variants like SwiGLU outperform standard activations by dynamically modulating feature flow.

Math & Tradeoffs→
M3-017Deep Learning FoundationsActivation Functions
Hard

How Activation Function Choice Impacts Numerical Stability and Training Speed

⚡ One-Sentence Answer

Activations determine gradient flow bounds, floating-point dynamic ranges, and GPU kernel memory access patterns, directly dictating convergence stability and hardware throughput.

Math & Tradeoffs→
M3-018Deep Learning FoundationsActivation Functions
Medium

How Activation Smoothness Impacts INT8/INT4 Quantization Accuracy

⚡ One-Sentence Answer

ReLU has exact zero cutoffs and bounded positive ranges making it quantization-friendly; smooth activations like GELU and SiLU produce heavy-tailed outlier activations that complicate low-bit quantization.

Math & Tradeoffs→
M3-019Deep Learning FoundationsActivation Functions
Medium

GELU vs SiLU: Differences, Connections, and Industry Model Adoption

⚡ One-Sentence Answer

Both are smooth self-gated activations; GELU gates via Gaussian CDF, while SiLU gates via Sigmoid; GELU is standard in BERT and GPT, while SiLU (SwiGLU) powers LLaMA and PaLM.

Math & Tradeoffs→
M3-020Deep Learning FoundationsNormalization Techniques
Easy

Comparing BatchNorm and LayerNorm: Normalization Dimensions and Applicable Regimes

⚡ One-Sentence Answer

BatchNorm normalizes across batch and spatial dimensions per channel (CNNs); LayerNorm normalizes across feature channels per individual sample (Transformers/RNNs).

Math & Tradeoffs→
M3-021Deep Learning FoundationsNormalization Techniques
Easy

RMSNorm vs LayerNorm: Mathematical Differences and Why LLaMA Adopted RMSNorm

⚡ One-Sentence Answer

RMSNorm eliminates mean-centering and learnable bias, scaling solely by the root-mean-square; it achieves identical modeling performance while cutting compute latency by 10–50%.

Math & Tradeoffs→
M3-022Deep Learning FoundationsNormalization Techniques
Medium

Pre-LN vs Post-LN in Transformers: Structural Differences and Gradient Highway Stability

⚡ One-Sentence Answer

Post-LN normalizes after the residual addition, causing vanishing/exploding gradients in deep layers; Pre-LN normalizes inputs inside the sub-layer, creating an unobstructed gradient highway.

Math & Tradeoffs→
M3-023Deep Learning FoundationsNormalization Techniques
Medium

Why BatchNorm Uses Running Statistics During Inference

⚡ One-Sentence Answer

Inference often runs on single samples (batch size = 1) where batch variance is zero, and predictions must be deterministic and independent of other co-batched inputs.

Math & Tradeoffs→
M3-024Deep Learning FoundationsNormalization Techniques
Hard

Side Effects of Normalization Layers and Alternative Architectures

⚡ One-Sentence Answer

Normalization layers introduce batch dependencies (BN), inference latency, and variance distortion; alternatives include Weight Standardization, Normalizer-Free networks (NFNet), and Fixup.

Math & Tradeoffs→
M3-025Deep Learning FoundationsNormalization Techniques
Medium

Group Normalization (GN): Mechanism and Applicable Scenarios

⚡ One-Sentence Answer

Group Normalization divides channels into groups and normalizes within each group per sample; being batch-independent, it is ideal for memory-heavy vision tasks like object detection.

Math & Tradeoffs→
M3-026Deep Learning FoundationsNormalization Techniques
Hard

Weight Standardization and Its Role in Normalizer-Free Networks

⚡ One-Sentence Answer

Weight Standardization normalizes convolutional weights across input channels and kernel spatial dimensions; paired with GroupNorm, it matches BatchNorm without batch dependencies.

Math & Tradeoffs→
M3-027Deep Learning FoundationsNormalization Techniques
Hard

QK-Norm and Normalization Techniques in Transformer Attention

⚡ One-Sentence Answer

QK-Norm applies LayerNorm or RMSNorm to Query and Key projections prior to computing dot-products, preventing attention logit growth, entropy collapse, and training divergence in large models.

Math & Tradeoffs→
M3-028Deep Learning FoundationsNormalization Techniques
Hard

DeepNorm: Scaling Transformers to 1,000+ Layers with Theoretical Stability

⚡ One-Sentence Answer

DeepNorm scales residual connections and downscales parameter initializations by layer depth, bounding activation variance growth to stabilize 1000-layer Transformers.

Math & Tradeoffs→
M3-029Deep Learning FoundationsNormalization Techniques
Hard

Normalization Operator Fusion in Inference and Its Impact on Serving Latency

⚡ One-Sentence Answer

Linear normalization (BatchNorm) folds directly into preceding convolutions with zero runtime overhead; non-linear norms (LayerNorm/RMSNorm) fuse with GEMM kernels to eliminate HBM roundtrips.

Math & Tradeoffs→
M3-030Deep Learning FoundationsOptimizers & Second-Order Methods
Easy

Update Formulas for SGD, SGD with Momentum, and Adam

⚡ One-Sentence Answer

SGD updates along negative gradient; Momentum adds velocity to escape saddle points; Adam normalizes updates using exponentially decaying first and second gradient moments.

Math & Tradeoffs→
M3-031Deep Learning FoundationsOptimizers & Second-Order Methods
Easy

Adam vs AdamW: Structural Differences and Why AdamW is Mathematically Correct

⚡ One-Sentence Answer

Adam couples L2 regularization with adaptive gradient moments, dampening decay for high-gradient parameters; AdamW restores true weight decay by decoupling it from gradient normalization.

Math & Tradeoffs→
M3-032Deep Learning FoundationsOptimizers & Second-Order Methods
Medium

Role of the Epsilon Parameter in Adam and Common Settings in Large Models

⚡ One-Sentence Answer

Epsilon prevents division by zero and bounds the maximum effective step size ($\eta/\epsilon$); LLMs typically use $10^{-8}$, or $10^{-6}$ in lower-precision (FP16/BF16) training.

Math & Tradeoffs→