L3Easy★ Core Essential · Industrial BedrockPart L · Classical ML & Statistical Simulation

Logistic Regression with L2 Regularization

⚡Flashcard Drill➔

Binary Logistic Regression optimizing L2-regularized cross-entropy via convex gradient descent.

⏱️ Time: O(iters * N * D)
💾 Space: O(D)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class LogisticRegressionL2:
def __init__(self, lr: float = 0.1, lambda_reg: float = 0.01, iters: int = 300):
self.lr = lr
self.lambda_reg = lambda_reg
self.iters = iters
self.w = None
self.b = 0.0
def fit(self, X: np.ndarray, y: np.ndarray):
N, D = X.shape
self.w = np.zeros(D)
self.b = 0.0
for _ in range(self.iters):
# 1. 线性预测与数值安全 Sigmoid
z =
z_clipped = np.clip(z, -88.0, 88.0)
p = 1.0 / (1.0 + np.exp(-z_clipped))
# 2. 计算误差项
error = p - y # (N,)
# 3. 梯度计算 (带 L2 权重衰减,偏置 b 通常不正则化)
grad_w = (1.0 / N) * (X.T @ error) + self.lambda_reg * self.w
grad_b = (1.0 / N) * np.sum(error)
# 4. 步进
self.w -= self.lr * grad_w
self.b -= self.lr * grad_b
def predict_proba(self, X: np.ndarray) -> np.ndarray:
z = np.clip(X @ self.w + self.b, -88.0, 88.0)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why is MSE a poor loss choice for Logistic Regression compared to Cross-Entropy?
Next: Principal Component Analysis (PCA) →
💡

Core Mental Anchor / Mnemonic

Compute prediction error (p - y), update weights with L2 decay, never regularize bias b.

📐 Mathematical Derivation & Core Formula

P(y=1∣x)=σ(w⊤x+b),L=−1N∑i=1N[yiln⁡pi+(1−yi)ln⁡(1−pi)]+λ2∥w∥22P(y=1 \mid x) = \sigma(w^\top x + b), \quad \mathcal{L} = -\frac{1}{N}\sum_{i=1}^N \left[y_i \ln p_i + (1 - y_i) \ln (1 - p_i)\right] + \frac{\lambda}{2} \|w\|_2^2
### Log-Odds & Convex Negative Log-Likelihood Gradient
Modeling log-odds ln⁡p1−p=w⊤x+b\ln\frac{p}{1-p} = w^\top x + b and minimizing L2-regularized binary cross-entropy leverages σ′(z)=σ(z)(1−σ(z))\sigma'(z) = \sigma(z)(1 - \sigma(z)) to cancel denominator derivatives cleanly:
∇wL=1NX⊤(σ(Xw+b)−y)+λw\nabla_w \mathcal{L} = \frac{1}{N} X^\top (\sigma(Xw + b) - y) + \lambda w
The resulting objective is strictly convex (with λ>0\lambda > 0), ensuring a unique global minimum.

🔄 Tensor Dimensions & Shape Flow

X (N, D) -> z = Xw+b -> sigmoid p -> grad_w = (1/N)X^T(p-y) + lambda*w -> update w, b

🛡️ Numerical Stability & Pitfalls Checklist

  • Clip logits safely before exp/sigmoid and compute log-loss stably to prevent underflow/overflow.
  • Exclude the bias term b from L2 weight regularization so baseline class priors remain unbiased.

💻 Reference Code Implementation

import numpy as np

class LogisticRegressionL2:
    def __init__(self, lr: float = 0.1, lambda_reg: float = 0.01, iters: int = 300):
        self.lr = lr
        self.lambda_reg = lambda_reg
        self.iters = iters
        self.w = None
        self.b = 0.0
        
    def fit(self, X: np.ndarray, y: np.ndarray):
        N, D = X.shape
        self.w = np.zeros(D)
        self.b = 0.0
        
        for _ in range(self.iters):
            # 1. 线性预测与数值安全 Sigmoid
            z = X @ self.w + self.b
            z_clipped = np.clip(z, -88.0, 88.0)
            p = 1.0 / (1.0 + np.exp(-z_clipped))
            
            # 2. 计算误差项
            error = p - y  # (N,)
            
            # 3. 梯度计算 (带 L2 权重衰减,偏置 b 通常不正则化)
            grad_w = (1.0 / N) * (X.T @ error) + self.lambda_reg * self.w
            grad_b = (1.0 / N) * np.sum(error)
            
            # 4. 步进
            self.w -= self.lr * grad_w
            self.b -= self.lr * grad_b
            
    def predict_proba(self, X: np.ndarray) -> np.ndarray:
        z = np.clip(X @ self.w + self.b, -88.0, 88.0)
        return 1.0 / (1.0 + np.exp(-z))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
X = np.array([[1.0], [2.0], [-1.0], [-2.0]])
y = np.array([1.0, 1.0, 0.0, 0.0])
clf = LogisticRegressionL2(lr=0.5, iters=500)
clf.fit(X, y)
assert clf.predict_proba(np.array([[3.0]])) > 0.8
assert clf.predict_proba(np.array([[-3.0]])) < 0.2
print("✓ 逻辑回归二分类自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why is MSE a poor loss choice for Logistic Regression compared to Cross-Entropy?
Composing Sigmoid with MSE yields a non-convex loss whose gradient contains σ(z)(1−σ(z))\sigma(z)(1-\sigma(z)); when predictions are confidently wrong, Sigmoid saturation zeroes out the gradient. Cross-entropy cancels that factor cleanly, making gradients proportional to (y^−y)(\hat{y} - y) and restoring convexity.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Linear Regression: Closed-Form & Gradient DescentBack to Part L & All 105Next: Principal Component Analysis (PCA) →