⚡ LLMsID:
bpe-tokenizationTokenization: BPE / WordPiece
分词 BPE/WordPiece
🎯Core Definition
BPE (Byte Pair Encoding) is a subword tokenization algorithm: start from byte/character-level units, count co-occurrence of adjacent symbol pairs, merge the most frequent byte pair into a new vocabulary symbol each iteration until the target vocab size is reached; encoding greedily matches the longest subword. Mainstream LLMs (GPT/LLaMA/Qwen) use byte-level BPE with vocabularies of 30K-150K tokens.
💡Use Cases
the tokenization stage before pretraining and inference; covering multilingual text and code (byte-level encodes any UTF-8 with no OOV), controlling token counts within fixed context windows, and subword splitting for compounding/agglutinative languages (German, Chinese, ...).
⚡Key Problems Solved
word-level tokenization yields huge vocabs and OOV on new words; character-level splitting costs 4-6 tokens per word and wastes context window. BPE encodes any text with a 30K-150K vocab at ~1-2 tokens per English word, balancing vocab size, sequence length, and cross-lingual coverage.
🎯5 High-Frequency Exam Points
1
BPE training: starting unit, what each iteration does, stopping condition?
2
Why do LLM vocabs sit at 30K-150K? Trade-offs of larger/smaller vocabs?
3
How does byte-level BPE guarantee no OOV for arbitrary Unicode text?
4
Selection criteria of BPE vs WordPiece vs Unigram?
5
Tokenizer impact: average tokens per English word? Multilingual/code behavior?
📖 In-depth Guide:📄 tokenizer-and-sampling →
Updated 2026-08-12🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "Tokenization: BPE / WordPiece"
Single choice pitfall questions with instant feedback and mistake tracking.