H2Easy★ Core Essential · Industrial BedrockPart H · Optimizers & Distributed Systems

Learning Rate Scheduler: Linear Warmup & Cosine Decay

⚡Flashcard Drill➔

Linear warmup to protect early random weights followed by smooth cosine annealing toward minimum LR.

⏱️ Time: O(1)
💾 Space: O(1)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def get_cosine_schedule_with_warmup(
current_step: int,
total_steps: int,
warmup_steps: int,
lr_max: float = 1e-3,
lr_min: float = 1e-5
) -> float:
# 1. 预热阶段
if current_step < warmup_steps:
return
# 2. 超出总步数则保持最低学习率
if current_step >= total_steps:
return lr_min
# 3. 余弦退火阶段
progress = (current_step - warmup_steps) / max(1, total_steps - warmup_steps)
cosine_decay = 0.5 * (1.0 + np.cos(np.pi * progress))
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why has the Warmup-Stable-Decay (WSD) schedule gained popularity in LLM pretraining?
Next: Gradient Clipping by Global Norm →
💡

Core Mental Anchor / Mnemonic

Ramp linearly up to max LR during warmup, then decay smoothly along cosine to min LR.

📐 Mathematical Derivation & Core Formula

ηt={tTwarmηmax⁡t≤Twarmηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡(t−TwarmTmax⁡−Twarmπ))t>Twarm\eta_t = \begin{cases} \frac{t}{T_{\text{warm}}} \eta_{\max} & t \le T_{\text{warm}} \\ \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\left(\frac{t - T_{\text{warm}}}{T_{\max} - T_{\text{warm}}} \pi\right)\right) & t > T_{\text{warm}} \end{cases}
### Linear Warmup & Cosine Annealing Mechanics
1. Linear Warmup: Ramps the learning rate smoothly from 00 to ηmax⁡\eta_{\max} during noisy initial steps while Adam moments stabilize;
2. Cosine Decay: Anneals smoothly along a cosine curve toward ηmin⁡\eta_{\min}, spending mid-training on steady progress and flattening curvature at the end for fine convergence.

🔄 Tensor Dimensions & Shape Flow

current_step -> warmup / cosine branch -> scalar learning rate eta_t

🛡️ Numerical Stability & Pitfalls Checklist

  • Warmup typically spans ~1%–5% of total steps to let Adam second-moment statistics stabilize.
  • Clamp step progress after total_steps at lr_min so the cosine curve never turns back upward.

💻 Reference Code Implementation

import numpy as np

def get_cosine_schedule_with_warmup(
    current_step: int,
    total_steps: int,
    warmup_steps: int,
    lr_max: float = 1e-3,
    lr_min: float = 1e-5
) -> float:
    # 1. 预热阶段
    if current_step < warmup_steps:
        return float(current_step / max(1, warmup_steps)) * lr_max
    
    # 2. 超出总步数则保持最低学习率
    if current_step >= total_steps:
        return lr_min
        
    # 3. 余弦退火阶段
    progress = (current_step - warmup_steps) / max(1, total_steps - warmup_steps)
    cosine_decay = 0.5 * (1.0 + np.cos(np.pi * progress))
    return lr_min + (lr_max - lr_min) * float(cosine_decay)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
total, warm = 100, 10
lr_0 = get_cosine_schedule_with_warmup(0, total, warm, lr_max=1.0, lr_min=0.0)
lr_peak = get_cosine_schedule_with_warmup(warm, total, warm, lr_max=1.0, lr_min=0.0)
lr_end = get_cosine_schedule_with_warmup(total, total, warm, lr_max=1.0, lr_min=0.0)
assert lr_0 == 0.0, "第 0 步应为 0"
assert np.isclose(lr_peak, 1.0), "预热结束应达到峰值"
assert np.isclose(lr_end, 0.0), "终点应回落至最小值"
print("✓ 学习率余弦调度器自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why has the Warmup-Stable-Decay (WSD) schedule gained popularity in LLM pretraining?
Standard cosine decay locks in total step count Tmax⁡T_{\max} upfront. WSD holds a flat peak learning rate indefinitely and runs a short decay only at the end, enabling flexible training extension and continual checkpoints.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Adam & AdamW Optimizer from ScratchBack to Part H & All 105Next: Gradient Clipping by Global Norm →