Capacity Estimation & Latency Budgeting

Capacity Estimation & Latency Budgeting is the first-principles quantitative derivation of throughput, storage footprint, network bandwidth, hardware compute costs, and latency SLAs during System Design interviews and engineering capacity planning; it formalizes 5 core derivation equations: 1) Throughput: $\text{Average QPS} = \frac{\text{DAU} \times \text{Requests/User}}{86,400\text{s}}$, with $\text{Peak QPS} = (2 \sim 5) \times \text{Average QPS}$; 2) Storage: $\text{Daily Storage} = \text{DAU} \times \text{Events/User} \times \text{Bytes/Event}$, scaled across 5-year retention with 3x replication factor; 3) Network Bandwidth: $\text{Peak QPS} \times \text{Payload Bytes} \times 8\text{ bits}$; 4) GPU Compute Fleet: $\text{GPUs} = \lceil \frac{\text{Peak QPS} \times \text{Latency/Request}}{\text{BatchConcurrency/GPU}} \rceil$; 5) Latency SLA decomposition allocating strict millisecond slices across Network RTT, Feature Store IO, Neural Forward Passes, and Serialization.

Quantitative phase of ML system design interviews, cloud infrastructure cost budgeting, and microservice SLA contract definitions.

Qualitative designs without numbers collapse in production (e.g. proposing a single PostgreSQL node for a 10M QPS streaming event ingest); rigorous back-of-the-envelope calculations mathematically justify component selection (Redis vs S3).

Derive average QPS, peak QPS (3x), and egress bandwidth for a 100M DAU service with 100 feed requests per user per day?

Calculate the exact RAM footprint for 100 million 1024-dim FP32 vector embeddings indexed with HNSW (including 1.5x graph overhead)?

Calculate how many H100 GPUs are required to serve 1,000 concurrent streaming LLM sessions at 50 tokens/sec throughput?

Apply foundational latency numbers every programmer should know (L1 Cache 0.5ns, RAM 100ns, SSD 100µs, Datacenter RTT 0.5ms) during design derivations?

Design a tiered storage cost optimization architecture: 7-day hot data on NVMe SSD vs cold archive on S3 with 3x replication factor?