M6-095M6: Multimodal & Generative ModelsVideo, 3D & Audio Generative ModelsMedium
⚡Flashcard Drill➔Mastery:
Foundational Pipelines of Speech Recognition (ASR) and Text-to-Speech (TTS)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Speech processing spans dual inverse workflows: Automatic Speech Recognition (ASR) decodes raw audio waveforms into text tokens via acoustic encoders, while Text-to-Speech (TTS) synthesizes audio waveforms from text via acoustic models and neural vocoders.
📌 Key Takeaways
- •ASR pipeline architecture: transforms raw audio into 80-channel log-mel spectrograms, encodes acoustic features via Conformer/Transformer backbones, and decodes text using CTC or autoregressive sequence-to-sequence
- •TTS two-stage architecture: Stage 1 (Acoustic Model) maps phoneme sequences to intermediate mel-spectrograms; Stage 2 (Neural Vocoder) synthesizes raw audio waveforms from spectrograms
- •Modern end-to-end tokenization: frontier omni-models replace multi-stage ASR/TTS pipelines with discrete neural audio codecs (EnCodec, SoundStream) unified under single autoregressive LLMs
📐 Mathematical Derivations
Mathematical Mechanism: 1. Automatic Speech Recognition (ASR) Pipeline (Whisper / Conformer): (a) Audio Preprocessing: 16kHz audio waveform is transformed via Short-Time Fourier Transform (STFT) with 25ms window and 10ms hop size into 80-channel log-mel spectrogram: (b) Acoustic Encoder: Maps to contextual features . (c) CTC / Autoregressive Decoding: In Connectionist Temporal Classification (CTC): In Whisper: Decodes text autoregressively conditioned on via cross-attention: 2. Text-to-Speech (TTS) Pipeline: (a) Text to Phonemes / Tokens: Text normalized and mapped to phonemes: . (b) Acoustic Model (FastSpeech / VITS): Predicts target mel-spectrogram conditioned on speaker embedding and duration/pitch predictors. (c) Neural Vocoder (HiFi-GAN / BigVGAN): Generates raw time-domain waveform from mel-spectrogram using transposed convolutions and multi-period discriminator: 3. Discrete Audio Codecs (EnCodec / Descript Audio Codec): Compresses audio into discrete multi-codebook tokens using Residual Vector Quantization (RVQ): .
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Phoneme vs Character Input in TTS: English orthography is notoriously irregular ('read' vs 'read', 'colonel' vs 'kernel'). Early TTS systems required complex grapheme-to-phoneme (G2P) dictionaries to avoid pronunciation errors. Modern large-scale TTS models (VALL-E, Voicebox) trained on 50,000+ hours of audio learn direct grapheme-to-speech mapping, eliminating brittle rule-based G2P frontends. ② Latency in Real-Time Conversational Speech (Turn-by-Turn vs Streaming): Cascading independent ASR + LLM + TTS introduces high compound latency: 500ms (ASR) + 800ms (LLM TTFT) + 400ms (TTS generation) = 1.7+ seconds, feeling unnatural to human conversational cadence. Frontier real-time voice models (GPT-4o) eliminate intermediate text serialization, streaming speech tokens end-to-end with sub-300ms latency. ③ Neural Vocoder Generalization: A vocoder trained strictly on clean studio speech produces harsh robotic squeaks when applied to outdoor or noisy spectrograms. Modern vocoders (BigVGAN) train on diverse noisy speech datasets using Snake activation functions to ensure artifact-free waveform reconstruction across arbitrary acoustic environments. ⑤ Interview Strategy: Diagram ASR (waveform mel encoder text) vs TTS (text acoustic model mel vocoder), formulate CTC alignment vs autoregressive decoding, explain why G2P dictionaries were historically necessary, and explain how neural audio codecs unify speech into LLM tokens.
⚠️ Common Interview Pitfalls
- ✕Attempting to generate raw audio waveforms directly with standard text LLMs without an acoustic model or neural vocoder
- ✕Ignoring the grapheme-to-phoneme (G2P) pronunciation challenges in legacy TTS architectures without massive pre-training data
- ✕Cascading separate ASR, LLM, and TTS pipelines in interactive voice agents without accounting for compound latency accumulation (> 1.5s)
🎯 Interviewer Follow-up Inquiries
- ?Why is Connectionist Temporal Classification (CTC) loss mathematically capable of aligning variable-length audio frames to shorter text token sequences?
- ?How does Residual Vector Quantization (RVQ) in neural audio codecs compress high-fidelity continuous audio into discrete token streams?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚