🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 938
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsGlobal System
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭Math Foundations Mindmap & Cards→📖Bayesian Inference, Entropy & KL Divergence→🎯DS Track→
Found 24 in-depth questions
M4-076Sequences & TransformersMixture of Experts (MoE)
Medium

Communication Bottlenecks and Expert Parallelism in MoE

⚡ One-Sentence Answer

Expert Parallelism distributes different experts across separate GPUs, requiring high-frequency all-to-all communication to dispatch and combine tokens that can easily bottleneck training and inference on low-bandwidth interconnects.

Math & Tradeoffs→
M4-077Sequences & TransformersMixture of Experts (MoE)
Medium

Inference Characteristics: Dense Models vs. MoE

⚡ One-Sentence Answer

MoE models match the fast generation FLOPs of small dense models during inference, but demand the massive VRAM footprint and memory bandwidth of huge models, creating distinct operational trade-offs across batch sizes.

Math & Tradeoffs→
M4-078Sequences & TransformersMixture of Experts (MoE)
Hard

Fine-Tuning and Deployment Challenges of MoE Models

⚡ One-Sentence Answer

MoE fine-tuning and deployment are complicated by routing instability on small task-specific datasets, high distributed memory requirements, and fragmented kernel execution during serving.

Math & Tradeoffs→
M4-079Sequences & TransformersMixture of Experts (MoE)
Hard

Shared Experts and Fine-Grained Expert Architecture in DeepSeek-MoE

⚡ One-Sentence Answer

DeepSeek-MoE innovates MoE architecture by dividing coarse experts into fine-grained micro-experts and dedicating fixed shared experts to isolate common foundational knowledge from specialized routing.

Math & Tradeoffs→
M4-080Sequences & TransformersMixture of Experts (MoE)
Hard

MoE Routing Algorithms: Top-k, Expert Choice, and Soft Routing

⚡ One-Sentence Answer

MoE routing has evolved from token-choice Top-$k$ (tokens select experts) to Expert Choice (experts select tokens) and Soft Routing (fully differentiable linear mixtures), each balancing token dropping, load balancing, and autoregressive causality.

Math & Tradeoffs→
M4-081Sequences & TransformersState Space Models (Mamba / S4)
Easy

Fundamental Formulation of State Space Models (SSM)

⚡ One-Sentence Answer

State Space Models map a 1D continuous input signal to an output through a latent hidden state via linear ordinary differential equations, combining the parallel training of CNNs with the $O(1)$ recurrent inference of RNNs.

Math & Tradeoffs→
M4-082Sequences & TransformersState Space Models (Mamba / S4)
Easy

Selective State Space Models (Selective SSM) in Mamba

⚡ One-Sentence Answer

Mamba transforms classical linear time-invariant SSMs into content-dependent models by making the parameters $(B, C, \Delta)$ functions of the input token, utilizing a hardware-aware parallel associative scan to achieve linear-time computation without global convolutions.

Math & Tradeoffs→
M4-083Sequences & TransformersState Space Models (Mamba / S4)
Medium

Complexity and Expressive Capability: SSM vs. Self-Attention

⚡ One-Sentence Answer

Self-attention provides lossless historical recall via dynamic $O(L^2)$ pairwise interactions with an $O(L)$ expanding KV cache, whereas SSM offers linear $O(L)$ compute and $O(1)$ memory via state compression at the cost of an information retrieval bottleneck.

Math & Tradeoffs→
M4-084Sequences & TransformersState Space Models (Mamba / S4)
Medium

Design Motivations for Hybrid Architectures Combining Attention and SSM

⚡ One-Sentence Answer

Hybrid architectures interleave a small fraction of full attention layers with a majority of linear SSM layers to achieve the exact retrieval power of Transformers alongside the linear scaling and minimal KV cache footprint of State Space Models.

Math & Tradeoffs→
M4-085Sequences & TransformersState Space Models (Mamba / S4)
Hard

Mamba Hardware-Aware Implementation: Parallel Scan and Kernel Fusion

⚡ One-Sentence Answer

Mamba achieves linear-time sequence modeling by fusing parameter discretization and associative prefix scanning into a single GPU SRAM kernel, avoiding the memory bandwidth bottleneck of writing expanded intermediate states to HBM.

Math & Tradeoffs→
M4-086Sequences & TransformersState Space Models (Mamba / S4)
Hard

SSM Initialization and Numerical Stability: HiPPO Matrix and Discretization Precision

⚡ One-Sentence Answer

State Space Models rely on the HiPPO matrix initialization to maintain long-range memory and require double-precision discretization with logarithmic parameterization to prevent gradient explosion and exponential numerical collapse.

Math & Tradeoffs→
M4-087Sequences & TransformersGraph Neural Networks (GNN / GAT)
Easy

Message Passing Neural Network (MPNN) Framework in GNNs

⚡ One-Sentence Answer

The MPNN framework unifies graph neural networks into an iterative three-phase computation: edge message generation, neighborhood aggregation, and node state updating.

Math & Tradeoffs→
M4-088Sequences & TransformersGraph Neural Networks (GNN / GAT)
Easy

Comparison of Graph Convolutional Networks: GCN, GraphSAGE, and GAT

⚡ One-Sentence Answer

GCN uses fixed symmetric degree normalization, GraphSAGE introduces mini-batch neighborhood sampling with flexible aggregators, and GAT dynamically learns anisotropic attention weights across neighboring edges.

Math & Tradeoffs→
M4-089Sequences & TransformersGraph Neural Networks (GNN / GAT)
Medium

Oversmoothing and Oversquashing in Deep GNNs

⚡ One-Sentence Answer

Oversmoothing causes all node representations to converge to identical uniform embeddings as GNN depth increases, while oversquashing causes exponential information bottlenecks when compressing multi-hop tree neighborhoods into fixed-size node vectors.

Math & Tradeoffs→
M4-090Sequences & TransformersGraph Neural Networks (GNN / GAT)
Medium

Large-Scale Graph Training Strategies: Node Sampling, Layer Sampling, and Subgraph Partitioning

⚡ One-Sentence Answer

Scaling GNNs to billion-edge graphs overcomes the neighbor explosion problem using three primary paradigms: node-wise sampling (GraphSAGE), layer-wise sampling (FastGCN), and subgraph partitioning (Cluster-GCN / GraphSAINT).

Math & Tradeoffs→
M4-091Sequences & TransformersGraph Neural Networks (GNN / GAT)
Hard

Heterogeneous Graphs and Relational Graph Convolutional Networks (R-GCN)

⚡ One-Sentence Answer

Heterogeneous graphs model multiple distinct node and edge types, with Relational Graph Convolutional Networks (R-GCN) dedicating type-specific transformation matrices to each relation alongside basis- or block-diagonal regularization to control parameter explosion.

Math & Tradeoffs→
M4-092Sequences & TransformersGraph Neural Networks (GNN / GAT)
Hard

Relationship Between Graph Neural Networks and Transformers

⚡ One-Sentence Answer

A standard Transformer is a fully-connected Graph Neural Network where all tokens are nodes with dynamic attention-weighted edges, while Graph Transformers generalize this by combining sparse physical graph structure with global attention via structural and positional encodings.

Math & Tradeoffs→
M4-093Sequences & TransformersSequence Modeling Trade-offs
Easy

Inductive Biases in Sequence Modeling: RNN, CNN, and Transformer

⚡ One-Sentence Answer

RNNs enforce sequential Markovian temporal bias with $O(1)$ state updates, CNNs enforce local translation invariance and hierarchical receptive fields, while Transformers discard spatial priors for permutation equivariance, trading inductive bias for massive expressive capacity.

Math & Tradeoffs→
M4-094Sequences & TransformersSequence Modeling Trade-offs
Easy

When to Still Choose RNN/LSTM Over Transformers: Low-Latency Edge, Microcontrollers, and Infinite Streaming

⚡ One-Sentence Answer

RNNs and LSTMs remain the superior engineering choice in hard real-time streaming, edge microcontrollers with strictly bounded RAM, and low-compute embedded sensor environments where an expanding KV cache is prohibitive.

Math & Tradeoffs→
M4-095Sequences & TransformersSequence Modeling Trade-offs
Medium

Architecture Selection Guide for Long-Sequence Tasks

⚡ One-Sentence Answer

Architectural selection for long-sequence tasks depends on query access patterns: choose Full Attention + FlashAttention for precise needle retrieval, Hybrid Attention-SSM for balanced long-document generation, and RAG for cost-effective massive corpus QA.

Math & Tradeoffs→
M4-096Sequences & TransformersSequence Modeling Trade-offs
Medium

Architectural Requirements for Order-Sensitive vs. Order-Invariant Tasks

⚡ One-Sentence Answer

Order-sensitive tasks require explicit sequence inductive biases such as causal masking and positional encodings, whereas order-invariant tasks require permutation-equivariant networks like DeepSets or Set Transformers.

Math & Tradeoffs→
M4-097Sequences & TransformersSequence Modeling Trade-offs
Hard

Systematic Evaluation of Long-Context Models: Beyond Needle-in-a-Haystack to RULER and Reasoning

⚡ One-Sentence Answer

Evaluating long-context models requires moving beyond simple, easily saturated single-needle retrieval to synthetic stress benchmarks like RULER and real-world multi-document reasoning tasks that evaluate multi-hop variable tracking and global aggregation.

Math & Tradeoffs→
M4-098Sequences & TransformersSequence Modeling Trade-offs
Hard

Length Extrapolation vs. Length Generalization in Sequence Models

⚡ One-Sentence Answer

Length extrapolation refers to a model's ability to maintain low perplexity on sequences longer than observed during training, whereas length generalization refers to algorithmic reasoning ability that successfully executes on larger problem sizes.

Math & Tradeoffs→
M4-099Sequences & TransformersModel Compression & Distillation
Easy

Knowledge Distillation Framework and the Role of Temperature $T$

⚡ One-Sentence Answer

Knowledge distillation transfers representation knowledge from a large teacher model to a compact student model by matching soft probability distributions, with temperature $T$ controlling the smoothness and information density of the dark knowledge dark probabilities.

Math & Tradeoffs→