M9-015M9: Senior ML & AI System DesignLLM Serving, RAG & Agent SysDesignHard
⚡Flashcard Drill➔Mastery:

Design a Full-Duplex Sub-400ms Voice-to-Voice Agent with Instant Barge-In

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M9-016 →
📐 Mathematical Definition
VAD+ASR+LLM+Chunk TTS≤400ms∧30ms Barge-In Flush\text{VAD} + \text{ASR} + \text{LLM} + \text{Chunk TTS} \le 400\text{ms} \quad \land \quad \text{30ms Barge-In Flush}
⚡ One-Sentence Core Answer
Build natural sub-400ms voice agents over WebRTC with semantic turn detection, 30ms echo-canceled barge-in truncation, and clause-level streaming TTS synthesis.

📌 Key Takeaways

  • •
    L1 Strict Budget: natural conversation pauses require <=400ms audio turnaround across all hops
  • •
    L2 Acoustic Echo & Barge-In: AEC + VAD cancels in-flight LLM generation and flushes playback within 30ms
  • •
    L2 Semantic Turn Taking: tiny classifier distinguishes thoughtful pauses ('um...') from real turn endings
  • •
    L3 Synced Context Memory: truncate assistant history strictly to tokens actually played before user interruption

📐 Mathematical Derivations

Synthesize speech at micro-clause boundaries (~6–10 tokens) rather than waiting for full paragraphs so first audio frames start smoothly.

🏭 Production Trade-offs

Trigger low-latency conversational fillers ahead of multi-hundred-millisecond database tool lookups.
⚠️ Common Interview Pitfalls
  • ✕
    Using synchronous HTTP audio file uploads instead of full-duplex WebRTC UDP media
  • ✕
    Retaining unheard pre-generated LLM text in memory after barge-in interrupts
🎯 Interviewer Follow-up Inquiries
  • ?
    When do cascaded ASR+LLM+TTS pipelines outperform native end-to-end audio transformers?
  • ?
    How does speaker diarization or energy gating block bystander noise from triggering false barge-in?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM9-014: Design a Sub-Second Ephemeral MicroVM Sandbox Platform for AI Coding Agents📋 Back to BankNext →M9-016: Design a Zero-Skew Online/Offline Feature Store with Point-in-Time Travel