M9-015M9: Senior ML & AI System DesignLLM Serving, RAG & Agent SysDesignHard
⚡Flashcard Drill➔Mastery:
Design a Full-Duplex Sub-400ms Voice-to-Voice Agent with Instant Barge-In
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Build natural sub-400ms voice agents over WebRTC with semantic turn detection, 30ms echo-canceled barge-in truncation, and clause-level streaming TTS synthesis.
📌 Key Takeaways
- •L1 Strict Budget: natural conversation pauses require <=400ms audio turnaround across all hops
- •L2 Acoustic Echo & Barge-In: AEC + VAD cancels in-flight LLM generation and flushes playback within 30ms
- •L2 Semantic Turn Taking: tiny classifier distinguishes thoughtful pauses ('um...') from real turn endings
- •L3 Synced Context Memory: truncate assistant history strictly to tokens actually played before user interruption
📐 Mathematical Derivations
Synthesize speech at micro-clause boundaries (~6–10 tokens) rather than waiting for full paragraphs so first audio frames start smoothly.
🏭 Production Trade-offs
Trigger low-latency conversational fillers ahead of multi-hundred-millisecond database tool lookups.
⚠️ Common Interview Pitfalls
- ✕Using synchronous HTTP audio file uploads instead of full-duplex WebRTC UDP media
- ✕Retaining unheard pre-generated LLM text in memory after barge-in interrupts
🎯 Interviewer Follow-up Inquiries
- ?When do cascaded ASR+LLM+TTS pipelines outperform native end-to-end audio transformers?
- ?How does speaker diarization or energy gating block bystander noise from triggering false barge-in?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚