🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 938
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsGlobal System
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭Math Foundations Mindmap & Cards→📖Bayesian Inference, Entropy & KL Divergence→🎯DS Track→
Found 24 in-depth questions
M4-028Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)
Medium

Context Length Extension in RoPE: Position Interpolation, NTK-Aware Scaling, and YaRN

⚡ One-Sentence Answer

Position Interpolation compresses positions linearly; NTK-Aware scaling scales low frequencies while preserving high-frequency resolution; YaRN adds temperature and variance corrections for stable extrapolation.

Math & Tradeoffs→
M4-029Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)
Hard

Position Interpolation (PI) for RoPE Extension and Its Trade-offs

⚡ One-Sentence Answer

Position Interpolation scales positions linearly ($p \to p/s$), mapping long context into the pretrained range; its primary cost is blurring high-frequency local resolution.

Math & Tradeoffs→
M4-030Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)
Hard

Why Models Can Infer Position Without Explicit Positional Encodings (NoPE)

⚡ One-Sentence Answer

Causal attention masks, sequence boundary tokens (BOS), and padding asymmetries break permutation invariance, implicitly leaking absolute and relative position information.

Math & Tradeoffs→
M4-031Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)
Medium

RoPE Implementation Details: Complex Formulation, Fast Kernel Tricks, and Base Frequencies

⚡ One-Sentence Answer

RoPE factorizes high dimensions into 2D orthogonal rotation pairs; efficient CUDA kernels apply element-wise multiplications, and scaling base frequency $\theta$ enables context window expansion.

Math & Tradeoffs→
M4-032Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)
Medium

ALiBi (Attention with Linear Biases): Mechanisms and Zero-Shot Length Extrapolation

⚡ One-Sentence Answer

ALiBi injects non-learned linear distance penalties directly into attention logits, allowing models trained on 2K tokens to extrapolate zero-shot to 8K+ tokens with stable perplexity.

Math & Tradeoffs→
M4-033Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)
Hard

2D and 3D Rotary Position Embeddings (RoPE) for Vision and Video Multimodal Models

⚡ One-Sentence Answer

2D/3D RoPE decomposes channel dimensions into spatial (height, width) and temporal (time) sub-bands, rotating each sub-band by its corresponding coordinate to preserve multi-dimensional geometry.

Math & Tradeoffs→
M4-034Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)
Hard

Benchmarking Positional Encodings and Context Length Extrapolation

⚡ One-Sentence Answer

Evaluation requires combining continuous language modeling perplexity across sequence lengths with fine-grained retrieval benchmarks like Needle In A Haystack (NIAH).

Math & Tradeoffs→
M4-035Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Easy

Multi-Query Attention (MQA) and Grouped-Query Attention (GQA): Slashing KV-Cache Memory

⚡ One-Sentence Answer

MQA shares a single Key/Value head across all Query heads; GQA groups Query heads to share Key/Value heads, slashing inference KV-cache by $4-8\times$ with zero quality degradation.

Math & Tradeoffs→
M4-036Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Easy

Multi-Head Latent Attention (MLA): DeepSeek's Low-Rank KV Cache Compression

⚡ One-Sentence Answer

MLA projects Key and Value vectors into a single low-dimensional latent vector via low-rank compression; caching only the compressed vector slashes KV cache by over 90% while outperforming MHA.

Math & Tradeoffs→
M4-037Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Medium

Sparse Attention Mechanisms and Sliding Window Attention (SWA)

⚡ One-Sentence Answer

Sparse attention restricts tokens to attending to local sliding windows, dilated strides, or global anchor tokens, cutting $O(N^2)$ complexity to $O(N \cdot W)$ linear time.

Math & Tradeoffs→
M4-038Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Medium

Linear Attention and Kernel Methods: Associative Matrix Multiplication and O(N) Complexity

⚡ One-Sentence Answer

Linear attention expresses softmax as an inner product of feature maps $\phi(q)^T \phi(k)$, allowing matrix multiplications to associate as $Q(K^T V)$ to slash complexity from $O(N^2)$ to $O(N d^2)$.

Math & Tradeoffs→
M4-039Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Hard

Differential Attention (Diff Transformer): Design Motivation and Noise Cancellation

⚡ One-Sentence Answer

Differential Attention computes the difference between two separate softmax attention maps ($A_1 - \lambda A_2$), canceling common-mode background noise and attention sinks to amplify true signals.

Math & Tradeoffs→
M4-040Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Hard

The Attention Sink Phenomenon: Origins, Mechanics, and StreamingLLM

⚡ One-Sentence Answer

Softmax requires attention weights to sum to 1.0, forcing models to dump excess probability mass onto initial tokens (BOS) as an information sink; preserving initial tokens enables infinite-length streaming.

Math & Tradeoffs→
M4-041Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Medium

Uptraining Multi-Head Attention into MQA/GQA: Initialization and Quality Recovery

⚡ One-Sentence Answer

Uptraining initializes MQA/GQA key-value weights by mean-pooling existing MHA heads, recovering 100% of original model performance with only ~5% of pretraining compute.

Math & Tradeoffs→
M4-042Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Hard

Native Sparse Attention (NSA): Multi-Branch Hardware-Aligned Sparse Design

⚡ One-Sentence Answer

NSA combines token compression, top-$k$ block selection, and local sliding windows into three parallel branches, achieving hardware-aligned linear acceleration on long contexts.

Math & Tradeoffs→
M4-043Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Medium

Cross-Attention vs Self-Attention: Structural Differences and Multimodal Roles

⚡ One-Sentence Answer

Self-attention derives Query, Key, and Value from the same sequence; Cross-attention derives Query from the target sequence and Key/Value from an external conditioning source.

Math & Tradeoffs→
M4-044Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Hard

Attention Entropy Collapse and How QK-Norm Resolves It

⚡ One-Sentence Answer

Entropy collapse occurs when unconstrained query-key magnitudes cause attention logits to explode, collapsing softmax into rigid one-hot distributions; QK-Norm bounds logits to preserve entropy.

Math & Tradeoffs→
M4-045Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Hard

Alternatives to Softmax Attention: Sigmoid Attention and Normalization Variants

⚡ One-Sentence Answer

Sigmoid attention decouples token competition and eliminates attention sinks; Softmax1 adds a dummy constant to the denominator to allow complete non-attention.

Math & Tradeoffs→
M4-046Sequences & TransformersAttention Variants (MHA / MQA / GQA)
Hard

Fusing Attention with External Memory and Retrieval: kNN-LM and RETRO

⚡ One-Sentence Answer

kNN-LM interpolates softmax output with nearest-neighbor vector retrieval; RETRO integrates retrieved text chunks directly into cross-attention blocks.

Math & Tradeoffs→
M4-047Sequences & TransformersEfficient Attention & FlashAttention
Easy

FlashAttention Core Philosophy: IO-Awareness, SRAM Tiling, and Memory Hierarchy

⚡ One-Sentence Answer

FlashAttention treats GPU High Bandwidth Memory (HBM) traffic as the primary bottleneck, using SRAM block tiling and online softmax to compute exact attention without saving $N \times N$ matrices.

Math & Tradeoffs→
M4-048Sequences & TransformersEfficient Attention & FlashAttention
Easy

Online Softmax: Enabling Block-by-Block Incremental Attention Tiling

⚡ One-Sentence Answer

Online softmax updates running maximums and normalizers dynamically as blocks arrive, eliminating the traditional requirement of scanning the full sequence before computing exponents.

Math & Tradeoffs→
M4-049Sequences & TransformersEfficient Attention & FlashAttention
Medium

FlashAttention Backward Pass: Recomputing Softmax in SRAM Without Saving Intermediate Matrices

⚡ One-Sentence Answer

The backward pass saves only forward normalizers ($m, \ell$), recomputing the attention matrix on-the-fly in SRAM blocks to eliminate $O(N^2)$ activation memory completely.

Math & Tradeoffs→
M4-050Sequences & TransformersEfficient Attention & FlashAttention
Medium

PagedAttention: Eliminating KV-Cache Memory Fragmentation via Virtual Memory Paging

⚡ One-Sentence Answer

PagedAttention adapts OS virtual memory paging to LLM serving, storing non-contiguous KV-cache blocks in virtual pages to eliminate internal/external fragmentation and enable memory sharing.

Math & Tradeoffs→
M4-051Sequences & TransformersEfficient Attention & FlashAttention
Hard

Ring Attention and Sequence Parallelism for Near-Infinite Context Windows

⚡ One-Sentence Answer

Ring Attention circulates Key and Value blocks in a peer-to-peer logical ring across distributed GPUs, overlapping communication with block attention to scale context to millions of tokens.

Math & Tradeoffs→