🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 112
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsM4: Sequences & Transformers
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭LLM Master Knowledge Mindmap→📖Transformer Architecture & RoPE→🎯KV Cache & PagedAttention→
Found 24 in-depth questions
M4-073Sequences & TransformersLong Context Extensions & Scaling
Hard

Empirical Attention Patterns in Long Contexts: Attention Sinks and Local Windows

⚡ One-Sentence Answer

Attention in long contexts is dominated by two empirical phenomena: initial tokens act as attention sinks absorbing large baseline scores regardless of semantics, while intermediate layers concentrate probability mass heavily within local sliding windows.

Math & Tradeoffs→
M4-074Sequences & TransformersMixture of Experts (MoE)
Easy

Mixture of Experts (MoE) Architecture and Sparse Activation

⚡ One-Sentence Answer

Mixture of Experts replaces dense feed-forward networks with multiple parallel expert sub-networks, using a gating router to conditionally activate only the top-$k$ experts per token, vastly scaling parameter capacity without increasing compute FLOPs.

Math & Tradeoffs→
M4-075Sequences & TransformersMixture of Experts (MoE)
Easy

Load Balancing Loss in MoE and Why Routing Collapse Occurs

⚡ One-Sentence Answer

Load balancing loss enforces uniform token distribution across all experts by penalizing router gate variance, preventing rich-get-richer feedback loops that cause routing collapse into a pseudo-dense model.

Math & Tradeoffs→
M4-076Sequences & TransformersMixture of Experts (MoE)
Medium

Communication Bottlenecks and Expert Parallelism in MoE

⚡ One-Sentence Answer

Expert Parallelism distributes different experts across separate GPUs, requiring high-frequency all-to-all communication to dispatch and combine tokens that can easily bottleneck training and inference on low-bandwidth interconnects.

Math & Tradeoffs→
M4-077Sequences & TransformersMixture of Experts (MoE)
Medium

Inference Characteristics: Dense Models vs. MoE

⚡ One-Sentence Answer

MoE models match the fast generation FLOPs of small dense models during inference, but demand the massive VRAM footprint and memory bandwidth of huge models, creating distinct operational trade-offs across batch sizes.

Math & Tradeoffs→
M4-078Sequences & TransformersMixture of Experts (MoE)
Hard

Fine-Tuning and Deployment Challenges of MoE Models

⚡ One-Sentence Answer

MoE fine-tuning and deployment are complicated by routing instability on small task-specific datasets, high distributed memory requirements, and fragmented kernel execution during serving.

Math & Tradeoffs→
M4-079Sequences & TransformersMixture of Experts (MoE)
Hard

Shared Experts and Fine-Grained Expert Architecture in DeepSeek-MoE

⚡ One-Sentence Answer

DeepSeek-MoE innovates MoE architecture by dividing coarse experts into fine-grained micro-experts and dedicating fixed shared experts to isolate common foundational knowledge from specialized routing.

Math & Tradeoffs→
M4-080Sequences & TransformersMixture of Experts (MoE)
Hard

MoE Routing Algorithms: Top-k, Expert Choice, and Soft Routing

⚡ One-Sentence Answer

MoE routing has evolved from token-choice Top-$k$ (tokens select experts) to Expert Choice (experts select tokens) and Soft Routing (fully differentiable linear mixtures), each balancing token dropping, load balancing, and autoregressive causality.

Math & Tradeoffs→
M4-081Sequences & TransformersState Space Models (Mamba / S4)
Easy

Fundamental Formulation of State Space Models (SSM)

⚡ One-Sentence Answer

State Space Models map a 1D continuous input signal to an output through a latent hidden state via linear ordinary differential equations, combining the parallel training of CNNs with the $O(1)$ recurrent inference of RNNs.

Math & Tradeoffs→
M4-082Sequences & TransformersState Space Models (Mamba / S4)
Easy

Selective State Space Models (Selective SSM) in Mamba

⚡ One-Sentence Answer

Mamba transforms classical linear time-invariant SSMs into content-dependent models by making the parameters $(B, C, \Delta)$ functions of the input token, utilizing a hardware-aware parallel associative scan to achieve linear-time computation without global convolutions.

Math & Tradeoffs→
M4-083Sequences & TransformersState Space Models (Mamba / S4)
Medium

Complexity and Expressive Capability: SSM vs. Self-Attention

⚡ One-Sentence Answer

Self-attention provides lossless historical recall via dynamic $O(L^2)$ pairwise interactions with an $O(L)$ expanding KV cache, whereas SSM offers linear $O(L)$ compute and $O(1)$ memory via state compression at the cost of an information retrieval bottleneck.

Math & Tradeoffs→
M4-084Sequences & TransformersState Space Models (Mamba / S4)
Medium

Design Motivations for Hybrid Architectures Combining Attention and SSM

⚡ One-Sentence Answer

Hybrid architectures interleave a small fraction of full attention layers with a majority of linear SSM layers to achieve the exact retrieval power of Transformers alongside the linear scaling and minimal KV cache footprint of State Space Models.

Math & Tradeoffs→
M4-085Sequences & TransformersState Space Models (Mamba / S4)
Hard

Mamba Hardware-Aware Implementation: Parallel Scan and Kernel Fusion

⚡ One-Sentence Answer

Mamba achieves linear-time sequence modeling by fusing parameter discretization and associative prefix scanning into a single GPU SRAM kernel, avoiding the memory bandwidth bottleneck of writing expanded intermediate states to HBM.

Math & Tradeoffs→
M4-086Sequences & TransformersState Space Models (Mamba / S4)
Hard

SSM Initialization and Numerical Stability: HiPPO Matrix and Discretization Precision

⚡ One-Sentence Answer

State Space Models rely on the HiPPO matrix initialization to maintain long-range memory and require double-precision discretization with logarithmic parameterization to prevent gradient explosion and exponential numerical collapse.

Math & Tradeoffs→
M4-087Sequences & TransformersGraph Neural Networks (GNN / GAT)
Easy

Message Passing Neural Network (MPNN) Framework in GNNs

⚡ One-Sentence Answer

The MPNN framework unifies graph neural networks into an iterative three-phase computation: edge message generation, neighborhood aggregation, and node state updating.

Math & Tradeoffs→
M4-088Sequences & TransformersGraph Neural Networks (GNN / GAT)
Easy

Comparison of Graph Convolutional Networks: GCN, GraphSAGE, and GAT

⚡ One-Sentence Answer

GCN uses fixed symmetric degree normalization, GraphSAGE introduces mini-batch neighborhood sampling with flexible aggregators, and GAT dynamically learns anisotropic attention weights across neighboring edges.

Math & Tradeoffs→
M4-089Sequences & TransformersGraph Neural Networks (GNN / GAT)
Medium

Oversmoothing and Oversquashing in Deep GNNs

⚡ One-Sentence Answer

Oversmoothing causes all node representations to converge to identical uniform embeddings as GNN depth increases, while oversquashing causes exponential information bottlenecks when compressing multi-hop tree neighborhoods into fixed-size node vectors.

Math & Tradeoffs→
M4-090Sequences & TransformersGraph Neural Networks (GNN / GAT)
Medium

Large-Scale Graph Training Strategies: Node Sampling, Layer Sampling, and Subgraph Partitioning

⚡ One-Sentence Answer

Scaling GNNs to billion-edge graphs overcomes the neighbor explosion problem using three primary paradigms: node-wise sampling (GraphSAGE), layer-wise sampling (FastGCN), and subgraph partitioning (Cluster-GCN / GraphSAINT).

Math & Tradeoffs→
M4-091Sequences & TransformersGraph Neural Networks (GNN / GAT)
Hard

Heterogeneous Graphs and Relational Graph Convolutional Networks (R-GCN)

⚡ One-Sentence Answer

Heterogeneous graphs model multiple distinct node and edge types, with Relational Graph Convolutional Networks (R-GCN) dedicating type-specific transformation matrices to each relation alongside basis- or block-diagonal regularization to control parameter explosion.

Math & Tradeoffs→
M4-092Sequences & TransformersGraph Neural Networks (GNN / GAT)
Hard

Relationship Between Graph Neural Networks and Transformers

⚡ One-Sentence Answer

A standard Transformer is a fully-connected Graph Neural Network where all tokens are nodes with dynamic attention-weighted edges, while Graph Transformers generalize this by combining sparse physical graph structure with global attention via structural and positional encodings.

Math & Tradeoffs→
M4-093Sequences & TransformersSequence Modeling Trade-offs
Easy

Inductive Biases in Sequence Modeling: RNN, CNN, and Transformer

⚡ One-Sentence Answer

RNNs enforce sequential Markovian temporal bias with $O(1)$ state updates, CNNs enforce local translation invariance and hierarchical receptive fields, while Transformers discard spatial priors for permutation equivariance, trading inductive bias for massive expressive capacity.

Math & Tradeoffs→
M4-094Sequences & TransformersSequence Modeling Trade-offs
Easy

When to Still Choose RNN/LSTM Over Transformers: Low-Latency Edge, Microcontrollers, and Infinite Streaming

⚡ One-Sentence Answer

RNNs and LSTMs remain the superior engineering choice in hard real-time streaming, edge microcontrollers with strictly bounded RAM, and low-compute embedded sensor environments where an expanding KV cache is prohibitive.

Math & Tradeoffs→
M4-095Sequences & TransformersSequence Modeling Trade-offs
Medium

Architecture Selection Guide for Long-Sequence Tasks

⚡ One-Sentence Answer

Architectural selection for long-sequence tasks depends on query access patterns: choose Full Attention + FlashAttention for precise needle retrieval, Hybrid Attention-SSM for balanced long-document generation, and RAG for cost-effective massive corpus QA.

Math & Tradeoffs→
M4-096Sequences & TransformersSequence Modeling Trade-offs
Medium

Architectural Requirements for Order-Sensitive vs. Order-Invariant Tasks

⚡ One-Sentence Answer

Order-sensitive tasks require explicit sequence inductive biases such as causal masking and positional encodings, whereas order-invariant tasks require permutation-equivariant networks like DeepSets or Set Transformers.

Math & Tradeoffs→