N4Hard★ AI Infra Frontier · DistServe / MooncakePart N · Data Pipelines & Production SysOps
Prefill / Decode Disaggregated Serving Simulator
Simulate disaggregated Prefill (compute-bound) and Decode (memory-bandwidth-bound) pools with explicit KV-transfer latency accounting.
⏱️ Time:
O(R log R)💾 Space:
O(R)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
def simulate_pd_disaggregation(requests: list, p_tps: float = 8000.0, d_tps_per_slot: float = 60.0, kv_bw_tok_per_s: float = 40000.0) -> list:
"""
Simulate Prefill/Decode disaggregated LLM serving vs coupled interference.
requests: list of (arrival_ms, prompt_tokens, gen_tokens).
Returns per-request (ttft_ms, total_ms) preserving zero prefill interference on active decode stream.
"""
p_free_ms = 0.0
out = []
for arr_ms, p_tok, d_tok in sorted(requests, key=lambda x: x[0]):
p_start =
prefill_dur = (p_tok / p_tps) * 1000.0
kv_xfer_dur = (p_tok / kv_bw_tok_per_s) * 1000.0
p_end = p_start + prefill_dur
p_free_ms = p_end
ttft_ms = (p_end - arr_ms) + kv_xfer_dur
decode_dur = (d_tok / d_tps_per_slot) * 1000.0
total_ms =
out.append((round(ttft_ms, 2), round(total_ms, 2)))
return out
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Compare Chunked Prefill vs. full Prefill/Decode Disaggregation.
💡
Core Mental Anchor / Mnemonic
Decouple compute-heavy Prefill from bandwidth-heavy Decode; budget KV network xfer delay honestly.
📐 Mathematical Derivation & Core Formula
### Isolating Decode TPOT Jitter via Disaggregation
Co-located continuous batching suffers severe TPOT spikes whenever a 16k-token prefill monopolizes SMs. Splitting Prefill and Decode workers trades extra RDMA/NVLink KV transfer latency for clean SLO isolation and independent pool sizing.
Co-located continuous batching suffers severe TPOT spikes whenever a 16k-token prefill monopolizes SMs. Splitting Prefill and Decode workers trades extra RDMA/NVLink KV transfer latency for clean SLO isolation and independent pool sizing.
🔄 Tensor Dimensions & Shape Flow
requests -> Prefill worker queue -> KV RDMA/NVLink xfer -> isolated Decode slot stream
🛡️ Numerical Stability & Pitfalls Checklist
- Model KV-transfer duration explicitly; disaggregation degrades TTFT when network bandwidth lags HBM.
- Prefer chunked prefill co-location for short-context workloads lacking high-speed RDMA fabrics.
💻 Reference Code Implementation
def simulate_pd_disaggregation(requests: list, p_tps: float = 8000.0, d_tps_per_slot: float = 60.0, kv_bw_tok_per_s: float = 40000.0) -> list:
"""
Simulate Prefill/Decode disaggregated LLM serving vs coupled interference.
requests: list of (arrival_ms, prompt_tokens, gen_tokens).
Returns per-request (ttft_ms, total_ms) preserving zero prefill interference on active decode stream.
"""
p_free_ms = 0.0
out = []
for arr_ms, p_tok, d_tok in sorted(requests, key=lambda x: x[0]):
p_start = max(arr_ms, p_free_ms)
prefill_dur = (p_tok / p_tps) * 1000.0
kv_xfer_dur = (p_tok / kv_bw_tok_per_s) * 1000.0
p_end = p_start + prefill_dur
p_free_ms = p_end
ttft_ms = (p_end - arr_ms) + kv_xfer_dur
decode_dur = (d_tok / d_tps_per_slot) * 1000.0
total_ms = ttft_ms + decode_dur
out.append((round(ttft_ms, 2), round(total_ms, 2)))
return out
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
# Request 1 long prompt (8k tokens) + Request 2 short prompt: PD guarantees decode TPOT stability on D-pool
reqs = [(0.0, 8000, 100), (200.0, 500, 100)]
res = simulate_pd_disaggregation(reqs, p_tps=8000.0, d_tps_per_slot=50.0, kv_bw_tok_per_s=40000.0)
# Req 0: prefill=1000ms, kv=200ms -> ttft=1200ms, decode=2000ms -> total=3200ms
# Req 1: starts P at 1000ms, prefill=62.5ms, kv=12.5ms -> waits 800ms -> ttft=875.0ms
assert abs(res[0][0] - 1200.0) < 1e-6 and abs(res[0][1] - 3200.0) < 1e-6
assert abs(res[1][0] - 875.0) < 1e-6
print("✓ Prefill-Decode disaggregation simulator assertion passed")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Compare Chunked Prefill vs. full Prefill/Decode Disaggregation.
Chunked prefill keeps KV local without RDMA hardware at modest TPOT jitter cost; full PD disaggregation excels at massive scale (>32k contexts, separate SLOs per pool) using RDMA layer-wise streaming.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.