ASR (Whisper)

ASR maps an audio sequence $x = (x_1, \dots, x_T)$ (~100 frames/s) to text $y = (y_1, \dots, y_L)$; the core difficulty is length mismatch ($T \gg L$) and alignment learning. Two main routes:

CTC (Connectionist Temporal Classification): a blank symbol is introduced and the marginal probability is summed over all monotonic alignments that collapse to $y$ after removing blanks and merging repeats:

Assumes monotonic alignment, needs no frame labels, decodes with prefix beam search; frames are conditionally independent — weak at long-range language modeling, but naturally streaming-friendly.

Attention encoder-decoder (LAS / Whisper): at step $l$ the decoder attends over encoder outputs, learning soft alignment implicitly with an intrinsic LM prior:

Higher accuracy but slower autoregressive decoding and no explicit alignment; hybrid systems share an encoder and train CTC + attention jointly (e.g. ESPnet), getting both.

Whisper weak supervision: 680k hours of multilingual audio-video captions (no manual per-sentence transcription), with a unified multi-task output format — the token sequence starts with task directives (language tag, task=transcribe/translate, timestamps) and text is generated autoregressively; robust to accents, noise and far-field recording.

meeting/subtitle transcription, voice assistant input, audio-to-text RAG pipelines; interviews compare CTC vs attention decoding and Whisper's data strategy.

HMM-GMM needed expert features, lexicons and forced alignment; CTC removes frame-labeling via dynamic programming, attention removes monotonicity and the external LM, and Whisper's weak supervision cuts labeling cost by two orders of magnitude while gaining cross-language robustness — now the de facto standard.

Test Your Knowledge: Practice Questions for "ASR (Whisper)"

Single choice pitfall questions with instant feedback and mistake tracking.