M5-006M5: NLP & Large Language ModelsTokenization (BPE / WordPiece / Unigram)Hard
⚡Flashcard Drill➔Mastery:

Vocabulary Expansion Techniques and Considerations

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-007 →
📐 Mathematical Definition
enew∼N(μold,σold);then continue pretraining with lower lre_{\text{new}}\sim\mathcal{N}(\mu_{\text{old}},\sigma_{\text{old}});\qquad \text{then continue pretraining with lower lr}
⚡ One-Sentence Core Answer
Vocabulary expansion adds domain- or language-specific subwords to a pre-trained model by resizing the embedding and LM head matrices, requiring careful weight initialization and targeted continual pre-training to prevent catastrophic forgetting.

📌 Key Takeaways

  • •
    Motivation: adapting an existing pre-trained LLM (e.g., LLaMA-1/2 with poor Chinese support) to a new language or domain by adding 10k–30k10\text{k}\text{--}30\text{k} new subword tokens
  • •
    Matrix resizing: expands embedding matrix WE∈RV×dW_E \in \mathbb{R}^{V \times d} to WE′∈R(V+K)×dW_E' \in \mathbb{R}^{(V + K) \times d} and adjusts output LM head WheadW_{\text{head}}
  • •
    Initialization strategy: initialize new token embeddings by averaging the embeddings of their constituent old subwords rather than using random Gaussian initialization
  • •
    Continual pre-training: freeze base Transformer layers initially to align new embeddings, then unfreeze all layers with small learning rates

📐 Mathematical Derivations

Mathematical Mechanism: 1. Subword Compositional Initialization: Let new token tnewt_{\text{new}} be added to vocabulary. Decompose tnewt_{\text{new}} into its sub-tokens under the old tokenizer Told\mathcal{T}_{\text{old}}: tnew=[u1,u2,…,um]t_{\text{new}} = [u_1, u_2, \dots, u_m], where ui∈Voldu_i \in \mathcal{V}_{\text{old}}. Initialize the new embedding vector e(tnew)e(t_{\text{new}}) as the mean of its constituent old embeddings: e(tnew)=1m∑i=1meold(ui)e(t_{\text{new}}) = \frac{1}{m} \sum_{i=1}^m e_{\text{old}}(u_i) This places new tokens in the semantic neighborhood of their components, preventing massive gradient shock at step 0. 2. Output Head Initialization: Similarly, initialize the output projection vector whead(tnew)w_{\text{head}}(t_{\text{new}}) from the mean of constituent output vectors: whead(tnew)=1m∑i=1mwhead(ui)w_{\text{head}}(t_{\text{new}}) = \frac{1}{m} \sum_{i=1}^m w_{\text{head}}(u_i)

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Random vs Mean Initialization: Random Gaussian initialization causes severe gradient spikes on the first training step, as uninitialized embeddings produce chaotic output logits that propagate massive gradients backward through all layers. Mean subword initialization stabilizes loss from step 1. ② Staged Training Pipeline (Chinese-LLaMA Paradigm): - Stage 1 (Embedding Warmup): Freeze all Transformer attention and MLP layers; train only the newly added embedding and head weights (WE′,Whead′W_E', W_{\text{head}}') on domain text for a few billion tokens. - Stage 2 (Full Alignment): Unfreeze all model parameters, training with a small learning rate (e.g., 10%10\% of original pre-training LR) and a 15-20% replay mixture of original pre-training data to prevent catastrophic forgetting. ③ Tokenizer Training Mixture: When training the new expansion tokenizer, train it on a mixture of target language text and original English text to preserve high English compression ratios. ④ Modern Relevance: In modern frontier LLMs (LLaMA-3, Qwen-2), native vocabularies (128k+128\text{k}+) already provide rich multilingual coverage, reducing the necessity of manual post-hoc vocabulary expansion. ⑤ Interview Strategy: Describe the matrix resizing mechanics, derive the subword averaging initialization formula, and explain the two-stage training regimen (freeze base →\to unfreeze with replay).
⚠️ Common Interview Pitfalls
  • ✕
    Initializing new token embeddings with random Gaussian noise (causes massive gradient shock and destroys pre-trained representations)
  • ✕
    Continual pre-training without original-language replay data (leads to severe catastrophic forgetting of general reasoning)
  • ✕
    Unfreezing all Transformer layers immediately before new embeddings have aligned their representation magnitudes
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is freeze-base warmup necessary during the first stage of vocabulary expansion?
  • ?
    How does Chinese-LLaMA merge its expanded vocabulary with the original LLaMA tokenizer without collision?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-005: Tokenizer Challenges in Multilingual and Code Domains📋 Back to BankNext →M5-007: Causal Language Modeling (CLM) vs. Masked Language Modeling (MLM)