Pre-Training Objectives & Optimizer Details

The mainstream pretraining objective is Next-Token Prediction (autoregressive cross-entropy $L = -\sum \log P(x_t | x_{<t})$); code models often use FIM (Fill-in-the-Middle): the document is split into prefix/suffix/middle, and with ~50% probability the model predicts the middle from both sides, learning bidirectional context. The optimizer standard is AdamW (β₁=0.9, β₂=0.95, weight decay=0.1) with gradient clipping (global norm 1.0) and a warmup + cosine LR schedule; on a loss spike, training rolls back to the last checkpoint before the spike and resumes.

pretraining and continued pretraining, code LLM training, debugging divergence or exploding gradients.

pure next-token prediction cannot use right-side context, limiting code completion — FIM sees both sides and markedly boosts code continuation; β₂=0.95 decays the second-moment estimate faster than the standard 0.999, better matching LLM gradient scales, and wd=0.1's strong weight decay curbs overfitting; clipping at 1.0 prevents single-step gradient explosions; a loss spike can destroy hours of training, and the rollback strategy reduces the cost to re-running only the minutes before the spike.

Test Your Knowledge: Practice Questions for "Pre-Training Objectives & Optimizer Details"

Single choice pitfall questions with instant feedback and mistake tracking.