Decoding Strategies

Decoding strategies decide how to pick the next token from the vocabulary distribution during generation. Greedy takes the argmax; beam search keeps the top-$B$ paths; temperature rescales the logits ($T \to 0$ recovers greedy, $T > 1$ flattens the distribution for more randomness); top-k samples only among the $k$ highest-probability tokens; top-p (nucleus) samples from the smallest set whose cumulative probability exceeds $p$. Standard order: temperature on the logits first, then top-p truncation, then sampling.

conversational and creative generation wants diversity and fluency — sampling (top-p + temperature + top-k); translation, summarization, code want deterministic output — greedy or a small beam; to suppress low-probability tokens while keeping randomness, top-p adaptively truncates by distribution shape.

greedy argmax tends to repeat and loop with little diversity; pure sampling picks low-probability tokens, producing ungrammatical or factually wrong output. top-p adapts the candidate set to the distribution shape (large when flat, small when peaked), temperature controls logit sharpness, and the pair balances quality vs diversity; beam's $B$ parallel paths mitigate greedy myopia.

Role of temperature; what do T=0 and T→∞ degenerate to?

top-p vs top-k? Why is top-p more adaptive?

Why temperature before top-p? What if the order is swapped?

Drawbacks of greedy and beam? Effects of beam size?

Sampling vs deterministic decoding: suitable tasks and the trade-off?