Model Families & Evolution

Mainstream open-source families differ in architecture details and training recipes: LLaMA — BPE tokenizer with a 32K vocab (128K in LLaMA-3), SwiGLU + RMSNorm + RoPE, GQA from 70B up, trained on 15.6T tokens; Qwen — 152K vocab (efficient for Chinese, ~1.2 chars/token), tied embeddings, GQA standard, YARN for long context; DeepSeek-V3 — 671B total / 37B activated MoE, MLA low-rank KV compression, 128K vocab, trained on 14.8T tokens. Overall trend: exploding token budgets, larger vocabs, and MoE + long context as defaults.

model selection, interview comparisons, and tracking the technology evolution.

turning 'the model' into comparable engineering decisions — Qwen for Chinese-heavy workloads (better vocab and Chinese data share, lower per-token cost), DeepSeek for code/reasoning (MoE with 37B activated out of 671B gives more capacity per FLOP), LLaMA for general English ecosystems (richest tooling and resources); it also explains why the tokenizer drives cost and speed: a bigger vocab packs more info per token but costs more embedding/softmax parameters.

LLaMA-3 vs Qwen: tokenizer and positional-encoding differences?

What is DeepSeek-V3's MoE + MLA configuration?

How does vocab size affect Chinese token efficiency and inference cost?

What does the token-scaling trend (1.4T → 15.6T) imply?

How does each family support long context (YARN / positional surgery)?