TTS

TTS (text-to-speech) generates speech from text. Pipeline evolution: concatenative/parametric synthesis (HMM, vocoders like Griffin-Lim phase recovery) → two-stage neural TTS (Tacotron 2 acoustic model + WaveGlow/HiFi-GAN vocoder) → end-to-end (VITS, Vall-E, CosyVoice). The end-to-end representative VITS combines:

Variational inference (VAE-style): text goes through an encoder and stochastic duration predictor giving alignment $A$ and latent speech representation $z$ — $z$ is obtained by transforming text hidden states with a normalizing flow: $z = f_\theta(h, A)$. Training optimizes the variational lower bound of the acoustic likelihood:

where $p_\theta(x|z)$ is the HiFi-GAN decoder producing the waveform directly; the KL term pulls the flow prior $p_\theta(z|h,A)$ (invertible, exact log-likelihood) toward the posterior $q_\phi(z|x)$, and the flow's flexibility keeps the KL from degenerating.

Alignment: Monotonic Alignment Search (MAS) learns text-speech alignment implicitly in latent space at training — no external aligner needed.

Adversarial training: a multi-period discriminator scores the generated waveform (GAN loss), sharply improving naturalness and high-frequency detail.

Later directions: Vall-E uses audio codec tokens for zero-shot voice cloning; GPT-4o speaks directly with native audio tokens.

voice assistants, audiobooks/dubbing, short-video voiceover, real-time spoken dialogue; interviews drill VITS's ELBO and the flow's role.

two-stage pipelines lose detail in the intermediate mel representation, accumulate errors, and need a separate vocoder; VITS is single-stage end-to-end with flow prior + GAN output, regressing the waveform from text directly — near-human naturalness, faster inference, no external aligner or vocoder.

Test Your Knowledge: Practice Questions for "TTS"

Single choice pitfall questions with instant feedback and mistake tracking.