H2Easy★ Core Essential · Industrial BedrockPart H · Optimizers & Distributed Systems

Learning Rate Scheduler: Linear Warmup & Cosine Decay

Mastery Status

Industrial-grade implementation and mathematical foundations of Learning Rate Scheduler: Linear Warmup & Cosine Decay.

⏱️ Time Complexity: O(1)

💾 Space Complexity: O(1)

💡

Core Mental Anchor / Mnemonic

Master Learning Rate Scheduler: Linear Warmup & Cosine Decay: enforce numerical stability, check tensor shapes, and eliminate redundant memory allocations.

📐 Mathematical Derivation & Core Formula

\eta_t = \begin{cases} \frac{t}{T_{\text{warm}}} \eta_{\max} & t \le T_{\text{warm}} \\ \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\left(\frac{t - T_{\text{warm}}}{T_{\max} - T_{\text{warm}}} \pi\right)\right) & t > T_{\text{warm}} \end{cases}

### Mathematical Derivation & Theoretical Principles
Detailed first-principles formulation and architectural mechanics for Learning Rate Scheduler: Linear Warmup & Cosine Decay.

Refer to the LaTeX equation above for the core operator definition. The operator is designed to ensure strict numerical bounds, avoiding floating-point overflows and gradient anomalies.

🔄 Tensor Dimensions & Shape Flow

current_step -> 判断 warmup / cosine -> 输出当前标量学习率 eta_t

🛡️ Industrial Numerical Stability & Pitfalls

Ensure proper multi-dimensional tensor broadcasting and keepdims retention.
Enforce numerical guards (eps clamping and overflow thresholds) during exponentiation and division.
Verify train versus eval mode behavioral distinctions (e.g. frozen running statistics and dropout bypass).

💻 Industrial Code Implementation

import numpy as np

def get_cosine_schedule_with_warmup(
    current_step: int,
    total_steps: int,
    warmup_steps: int,
    lr_max: float = 1e-3,
    lr_min: float = 1e-5
) -> float:
    # 1. 预热阶段
    if current_step < warmup_steps:
        return float(current_step / max(1, warmup_steps)) * lr_max
    
    # 2. 超出总步数则保持最低学习率
    if current_step >= total_steps:
        return lr_min
        
    # 3. 余弦退火阶段
    progress = (current_step - warmup_steps) / max(1, total_steps - warmup_steps)
    cosine_decay = 0.5 * (1.0 + np.cos(np.pi * progress))
    return lr_min + (lr_max - lr_min) * float(cosine_decay)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter to verify correctness:

import numpy as np
total, warm = 100, 10
lr_0 = get_cosine_schedule_with_warmup(0, total, warm, lr_max=1.0, lr_min=0.0)
lr_peak = get_cosine_schedule_with_warmup(warm, total, warm, lr_max=1.0, lr_min=0.0)
lr_end = get_cosine_schedule_with_warmup(total, total, warm, lr_max=1.0, lr_min=0.0)
assert lr_0 == 0.0, "第 0 步应为 0"
assert np.isclose(lr_peak, 1.0), "预热结束应达到峰值"
assert np.isclose(lr_end, 0.0), "终点应回落至最小值"
print("✓ 学习率余弦调度器自测通过")

🎯 Core Architecture Follow-up Q&A

Q1:What are the key trade-offs and memory bottlenecks when deploying Learning Rate Scheduler: Linear Warmup & Cosine Decay in high-throughput inference?

Memory bandwidth (HBM to SRAM I/O) is the primary latency factor. Fusing element-wise operations and avoiding intermediate tensor materialization significantly outperforms naive implementations.

Q2:How does Learning Rate Scheduler: Linear Warmup & Cosine Decay handle extreme numerical boundaries or precision reduction (FP16/BF16/INT8)?

Under low precision, operations must be upcasted to FP32 during accumulation to prevent underflow/overflow, followed by proper scaling and clamping before converting back to the target format.

←Prev: Adam & AdamW Optimizer from Scratch All 69 Kernels Next: Gradient Clipping by Global Norm→