Audio Representation

Audio representation has three levels, from raw waveform to model-ready features, determining input dimension and information retained:

1. Waveform: 16 kHz mono 16-bit PCM is 16000 scalars per second at 256 kbps — lossless but extremely long sequences, expensive to model directly;

2. Mel-spectrogram: STFT (25 ms window, 10 ms hop) → mel filterbank (typically 80 bins) → log magnitude, yielding ~100 frames/s × 80 dims; keeps perceptual main information, drops phase, and is Whisper's input — but continuous, mismatched with text tokens, and phase must be restored by a vocoder;

3. Discrete audio tokens: neural codecs (EnCodec / SoundStream / DAC) compress the waveform through an encoder + RVQ residual quantization into discrete tokens: ~50-75 Hz frame rate, $K$ cascaded codebook tokens per frame, only a few kbps — roughly 1/50 to 1/100 of the raw 256 kbps (e.g. EnCodec 24 kHz audio @ 6-24 kbps).

Mel vs tokens: the spectrogram is dense and continuous but can't enter a discrete symbol space; tokens unify audio with text in one discrete space — the foundation of speech LLMs and end-to-end TTS.

ASR/TTS feature input (mel), speech LLMs and voice cloning (audio tokens), audio generation; interviews drill the dimensions/bitrates.

waveforms are too long and spectrograms non-discrete to unify with text; neural codecs shrink 16000 samples/s into a few hundred tokens/s, cutting bitrate by two orders of magnitude (256 kbps → a few kbps) while preserving semantics and speaker identity, letting LLMs and diffusion models consume audio directly.

How is a mel-spectrogram computed; what do 80 bins, 25 ms, 10 ms mean?

Pros and cons of waveform vs mel-spectrogram vs discrete tokens?

How do neural codecs tokenize audio; typical bitrate and compression ratio?

Why do speech LLMs consume discrete audio tokens instead of regressing mels?

Why does RVQ reach ~1/100 bitrate with acceptable reconstruction quality?