N4Hard★ AI Infra Frontier · DistServe / MooncakePart N · Data Pipelines & Production SysOps

Prefill / Decode Disaggregated Serving Simulator

⚡Flashcard Drill➔

Simulate disaggregated Prefill (compute-bound) and Decode (memory-bandwidth-bound) pools with explicit KV-transfer latency accounting.

⏱️ Time: O(R log R)
💾 Space: O(R)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
def simulate_pd_disaggregation(requests: list, p_tps: float = 8000.0, d_tps_per_slot: float = 60.0, kv_bw_tok_per_s: float = 40000.0) -> list:
"""
Simulate Prefill/Decode disaggregated LLM serving vs coupled interference.
requests: list of (arrival_ms, prompt_tokens, gen_tokens).
Returns per-request (ttft_ms, total_ms) preserving zero prefill interference on active decode stream.
"""
p_free_ms = 0.0
out = []
for arr_ms, p_tok, d_tok in sorted(requests, key=lambda x: x[0]):
p_start =
prefill_dur = (p_tok / p_tps) * 1000.0
kv_xfer_dur = (p_tok / kv_bw_tok_per_s) * 1000.0
p_end = p_start + prefill_dur
p_free_ms = p_end
ttft_ms = (p_end - arr_ms) + kv_xfer_dur
decode_dur = (d_tok / d_tps_per_slot) * 1000.0
total_ms =
out.append((round(ttft_ms, 2), round(total_ms, 2)))
return out
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Compare Chunked Prefill vs. full Prefill/Decode Disaggregation.
Next: Multi-Tenant Token Bucket with Pre-Reserve & Settlement →
💡

Core Mental Anchor / Mnemonic

Decouple compute-heavy Prefill from bandwidth-heavy Decode; budget KV network xfer delay honestly.

📐 Mathematical Derivation & Core Formula

LE2E=TTFTP+LKV_xfer+Nout⋅TPOTDL_{\text{E2E}} = \text{TTFT}_{\text{P}} + L_{\text{KV\_xfer}} + N_{\text{out}} \cdot \text{TPOT}_{\text{D}}
### Isolating Decode TPOT Jitter via Disaggregation
Co-located continuous batching suffers severe TPOT spikes whenever a 16k-token prefill monopolizes SMs. Splitting Prefill and Decode workers trades extra RDMA/NVLink KV transfer latency for clean SLO isolation and independent pool sizing.

🔄 Tensor Dimensions & Shape Flow

requests -> Prefill worker queue -> KV RDMA/NVLink xfer -> isolated Decode slot stream

🛡️ Numerical Stability & Pitfalls Checklist

  • Model KV-transfer duration explicitly; disaggregation degrades TTFT when network bandwidth lags HBM.
  • Prefer chunked prefill co-location for short-context workloads lacking high-speed RDMA fabrics.

💻 Reference Code Implementation

def simulate_pd_disaggregation(requests: list, p_tps: float = 8000.0, d_tps_per_slot: float = 60.0, kv_bw_tok_per_s: float = 40000.0) -> list:
    """
    Simulate Prefill/Decode disaggregated LLM serving vs coupled interference.
    requests: list of (arrival_ms, prompt_tokens, gen_tokens).
    Returns per-request (ttft_ms, total_ms) preserving zero prefill interference on active decode stream.
    """
    p_free_ms = 0.0
    out = []
    for arr_ms, p_tok, d_tok in sorted(requests, key=lambda x: x[0]):
        p_start = max(arr_ms, p_free_ms)
        prefill_dur = (p_tok / p_tps) * 1000.0
        kv_xfer_dur = (p_tok / kv_bw_tok_per_s) * 1000.0
        p_end = p_start + prefill_dur
        p_free_ms = p_end
        ttft_ms = (p_end - arr_ms) + kv_xfer_dur
        decode_dur = (d_tok / d_tps_per_slot) * 1000.0
        total_ms = ttft_ms + decode_dur
        out.append((round(ttft_ms, 2), round(total_ms, 2)))
    return out

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

# Request 1 long prompt (8k tokens) + Request 2 short prompt: PD guarantees decode TPOT stability on D-pool
reqs = [(0.0, 8000, 100), (200.0, 500, 100)]
res = simulate_pd_disaggregation(reqs, p_tps=8000.0, d_tps_per_slot=50.0, kv_bw_tok_per_s=40000.0)
# Req 0: prefill=1000ms, kv=200ms -> ttft=1200ms, decode=2000ms -> total=3200ms
# Req 1: starts P at 1000ms, prefill=62.5ms, kv=12.5ms -> waits 800ms -> ttft=875.0ms
assert abs(res[0][0] - 1200.0) < 1e-6 and abs(res[0][1] - 3200.0) < 1e-6
assert abs(res[1][0] - 875.0) < 1e-6
print("✓ Prefill-Decode disaggregation simulator assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Compare Chunked Prefill vs. full Prefill/Decode Disaggregation.
Chunked prefill keeps KV local without RDMA hardware at modest TPOT jitter cost; full PD disaggregation excels at massive scale (>32k contexts, separate SLOs per pool) using RDMA layer-wise streaming.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Population Stability Index (PSI) Drift MonitorBack to Part N & All 105Next: Multi-Tenant Token Bucket with Pre-Reserve & Settlement →