🔬 Applied Scientist & ML Specialist938 Problems · 127 Topics🌐 Linked to Tech Vault

TalentMe Science Depth: Universal Applied Science & ML Depth Problem Bank

938 end-to-end science depth, system design, transition bridge & BQ problems covering derivations, production trade-offs, pitfalls, and follow-ups.

🎯

Science Depth Mastery Tracker

0% Mastered

Locally synced in your browser. Track, review, and master all 938 scientific & design topics.

Mastered: 0
Reviewing: 0
Unstarted: 112
🚀 Start Next Unpracticed →
🌐
Linked to TalentMe Tech Vault & MindmapsM4: Sequences & Transformers
All 938 questions are directly linked to TalentMe's Tech Vault guides, atomic cards, and interactive mindmaps.
🧭LLM Master Knowledge Mindmap→📖Transformer Architecture & RoPE→🎯KV Cache & PagedAttention→
Found 24 in-depth questions
M4-049Sequences & TransformersEfficient Attention & FlashAttention
Medium

FlashAttention Backward Pass: Recomputing Softmax in SRAM Without Saving Intermediate Matrices

⚡ One-Sentence Answer

The backward pass saves only forward normalizers ($m, \ell$), recomputing the attention matrix on-the-fly in SRAM blocks to eliminate $O(N^2)$ activation memory completely.

Math & Tradeoffs→
M4-050Sequences & TransformersEfficient Attention & FlashAttention
Medium

PagedAttention: Eliminating KV-Cache Memory Fragmentation via Virtual Memory Paging

⚡ One-Sentence Answer

PagedAttention adapts OS virtual memory paging to LLM serving, storing non-contiguous KV-cache blocks in virtual pages to eliminate internal/external fragmentation and enable memory sharing.

Math & Tradeoffs→
M4-051Sequences & TransformersEfficient Attention & FlashAttention
Hard

Ring Attention and Sequence Parallelism for Near-Infinite Context Windows

⚡ One-Sentence Answer

Ring Attention circulates Key and Value blocks in a peer-to-peer logical ring across distributed GPUs, overlapping communication with block attention to scale context to millions of tokens.

Math & Tradeoffs→
M4-052Sequences & TransformersEfficient Attention & FlashAttention
Hard

Comparing Attention Optimization Paradigms: IO-Aware, Sparse, and Low-Rank/Approximation

⚡ One-Sentence Answer

IO-aware methods compute exact attention by optimizing hardware memory traffic (FlashAttention); Sparse methods enforce structural windows (SWA/NSA); Approximation methods linearize operations via low-rank kernels.

Math & Tradeoffs→
M4-053Sequences & TransformersEfficient Attention & FlashAttention
Medium

IO Complexity Analysis and the GPU Roofline Performance Model

⚡ One-Sentence Answer

The Roofline Model identifies whether a kernel is memory-bandwidth bound or compute bound by comparing its Arithmetic Intensity against the hardware balance point.

Math & Tradeoffs→
M4-054Sequences & TransformersEfficient Attention & FlashAttention
Hard

Architectural Innovations in FlashAttention-2 and FlashAttention-3

⚡ One-Sentence Answer

FlashAttention-2 optimizes outer-loop parallelism and warp partitioning ($2\times$ speedup); FlashAttention-3 leverages Hopper TMA, Tensor Core FP8, and asynchronous warp specialization ($1.5-2\times$ over FA2).

Math & Tradeoffs→
M4-055Sequences & TransformersEfficient Attention & FlashAttention
Hard

Decoding Phase Attention Bottlenecks and the Flash-Decoding Solution

⚡ One-Sentence Answer

Autoregressive decoding has a single query token ($N_q=1$) attending to a long KV cache, causing extreme GPU under-utilization; Flash-Decoding parallelizes over the sequence dimension to restore full GPU saturation.

Math & Tradeoffs→
M4-056Sequences & TransformersEfficient Attention & FlashAttention
Medium

Operator Fusion and torch.compile Speedups in Modern Attention Architectures

⚡ One-Sentence Answer

Operator fusion combines multiple memory-bound element-wise operations into a single GPU kernel, eliminating HBM roundtrips; `torch.compile` automates this via TorchDynamo and Triton code generation.

Math & Tradeoffs→
M4-057Sequences & TransformersKV Cache & Inference Optimizations
Easy

KV Cache Principles and Why It Cannot Be Used During Training

⚡ One-Sentence Answer

The KV Cache caches Key and Value representations of historical tokens during autoregressive inference to eliminate redundant $O(L^3)$ recomputation, whereas training processes the entire sequence simultaneously in parallel via causal masking without sequential dependencies.

Math & Tradeoffs→
M4-058Sequences & TransformersKV Cache & Inference Optimizations
Easy

Differences Between Prefill and Decode Phases in LLM Inference

⚡ One-Sentence Answer

The prefill phase processes the prompt in parallel and is compute-bound, whereas the decode phase generates tokens autoregressively one by one and is memory-bandwidth bound.

Math & Tradeoffs→
M4-059Sequences & TransformersKV Cache & Inference Optimizations
Medium

Losslessness and Theoretical Guarantee of Speculative Decoding

⚡ One-Sentence Answer

Speculative decoding uses a small draft model to generate candidate tokens and verifies them in parallel with the target model using modified rejection sampling, mathematically guaranteeing identical output distribution to the target model.

Math & Tradeoffs→
M4-060Sequences & TransformersKV Cache & Inference Optimizations
Medium

Continuous Batching vs. Static Batching in LLM Serving

⚡ One-Sentence Answer

Continuous batching operates at the iteration level to dynamically insert arriving requests and evict completed requests at every decoding step, eliminating the massive bubble overhead of static sequence-level batching.

Math & Tradeoffs→
M4-061Sequences & TransformersKV Cache & Inference Optimizations
Hard

Prefix Caching and RadixAttention Mechanisms

⚡ One-Sentence Answer

Prefix caching avoids redundant prefill computation by caching and reusing the KV tensors of shared prefix tokens across requests, with RadixAttention managing cached states hierarchically as a prefix tree.

Math & Tradeoffs→
M4-062Sequences & TransformersKV Cache & Inference Optimizations
Hard

Methods to Reduce KV Cache Memory Footprint

⚡ One-Sentence Answer

KV cache memory is minimized across four complementary dimensions: architecture design (MQA/GQA/MLA), numerical quantization (FP8/INT4), token pruning/eviction (H2O/StreamingLLM), and memory management (PagedAttention).

Math & Tradeoffs→
M4-063Sequences & TransformersKV Cache & Inference Optimizations
Hard

Chunked Prefill and Piggyback Hybrid Batch Scheduling

⚡ One-Sentence Answer

Chunked prefill splits lengthy prompt computations into fixed-size chunks and interleaves them with ongoing decode steps within the same batch iteration, preventing long prefills from causing decode latency spikes.

Math & Tradeoffs→
M4-064Sequences & TransformersKV Cache & Inference Optimizations
Hard

Speculative Decoding Variants: Medusa, EAGLE, and Multi-Token Prediction

⚡ One-Sentence Answer

Modern speculative decoding variants eliminate the need for an external draft model by using multi-head prediction (Medusa), feature-level autoregression (EAGLE), or pre-trained multi-token prediction heads (MTP).

Math & Tradeoffs→
M4-065Sequences & TransformersKV Cache & Inference Optimizations
Hard

Throughput vs. Latency Trade-offs in Inference: TTFT, TPOT, and Batch Size

⚡ One-Sentence Answer

LLM inference balances Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT) against global system throughput via batch size, navigating the Pareto frontier between compute saturation and memory bandwidth constraints.

Math & Tradeoffs→
M4-066Sequences & TransformersKV Cache & Inference Optimizations
Hard

Prefill-Decode Disaggregation Architecture

⚡ One-Sentence Answer

Prefill-Decode (PD) Disaggregation physically separates the prefill and decode phases onto dedicated heterogeneous GPU clusters, eliminating resource interference and enabling independent hardware scaling.

Math & Tradeoffs→
M4-067Sequences & TransformersLong Context Extensions & Scaling
Easy

Core Challenges of Scaling to Long Context Windows

⚡ One-Sentence Answer

Scaling to long contexts faces four fundamental bottlenecks: quadratic computational and linear KV memory growth, position encoding out-of-distribution failure, degradation of retrieval and reasoning attention, and high training data scarcity.

Math & Tradeoffs→
M4-068Sequences & TransformersLong Context Extensions & Scaling
Easy

The 'Lost in the Middle' Phenomenon in Long Context Models

⚡ One-Sentence Answer

Decoder-only models exhibit a U-shaped performance curve in long contexts, retrieving and reasoning over information at the very beginning and end of prompts with high accuracy while frequently ignoring information in the middle.

Math & Tradeoffs→
M4-069Sequences & TransformersLong Context Extensions & Scaling
Medium

Long Context Scaling Strategies: Length Extrapolation, RAG, and Architectural Redesign

⚡ One-Sentence Answer

Long context capabilities are tackled via three distinct paradigms: position extrapolation (RoPE scaling), Retrieval-Augmented Generation (RAG), and efficient architectural redesigns (linear attention/SSM/sparse attention).

Math & Tradeoffs→
M4-070Sequences & TransformersLong Context Extensions & Scaling
Medium

Hybrid Design of Sliding Window Attention and Global Tokens

⚡ One-Sentence Answer

Hybrid attention restricts the majority of tokens to attend within a local sliding window of size $W$ while allowing select global tokens to attend across the entire sequence, reducing complexity from $O(L^2)$ to $O(L \cdot W)$ while preserving long-range routing.

Math & Tradeoffs→
M4-071Sequences & TransformersLong Context Extensions & Scaling
Hard

Data Construction and Curriculum Learning for Long-Context Training

⚡ One-Sentence Answer

Long-context capability is most efficiently acquired by first pre-training on high-throughput short sequences, followed by multi-stage curriculum fine-tuning that progressively increases sequence length while balancing synthetic retrieval tasks and natural long documents.

Math & Tradeoffs→
M4-072Sequences & TransformersLong Context Extensions & Scaling
Hard

Context Length vs. Inference Cost and Economic Trade-offs

⚡ One-Sentence Answer

Long-context inference cost scales super-linearly due to prefill compute, massive KV cache memory footprint, and reduced serving concurrency, making selective context compression and hybrid RAG essential for production economics.

Math & Tradeoffs→