L3Easy★ Core Essential · Industrial BedrockPart L · Classical ML & Statistical Simulation
Logistic Regression with L2 Regularization
Binary Logistic Regression optimizing L2-regularized cross-entropy via convex gradient descent.
⏱️ Time:
O(iters * N * D)💾 Space:
O(D)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class LogisticRegressionL2:
def __init__(self, lr: float = 0.1, lambda_reg: float = 0.01, iters: int = 300):
self.lr = lr
self.lambda_reg = lambda_reg
self.iters = iters
self.w = None
self.b = 0.0
def fit(self, X: np.ndarray, y: np.ndarray):
N, D = X.shape
self.w = np.zeros(D)
self.b = 0.0
for _ in range(self.iters):
# 1. 线性预测与数值安全 Sigmoid
z =
z_clipped = np.clip(z, -88.0, 88.0)
p = 1.0 / (1.0 + np.exp(-z_clipped))
# 2. 计算误差项
error = p - y # (N,)
# 3. 梯度计算 (带 L2 权重衰减,偏置 b 通常不正则化)
grad_w = (1.0 / N) * (X.T @ error) + self.lambda_reg * self.w
grad_b = (1.0 / N) * np.sum(error)
# 4. 步进
self.w -= self.lr * grad_w
self.b -= self.lr * grad_b
def predict_proba(self, X: np.ndarray) -> np.ndarray:
z = np.clip(X @ self.w + self.b, -88.0, 88.0)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why is MSE a poor loss choice for Logistic Regression compared to Cross-Entropy?
💡
Core Mental Anchor / Mnemonic
Compute prediction error (p - y), update weights with L2 decay, never regularize bias b.
📐 Mathematical Derivation & Core Formula
### Log-Odds & Convex Negative Log-Likelihood Gradient
Modeling log-odds and minimizing L2-regularized binary cross-entropy leverages to cancel denominator derivatives cleanly:
The resulting objective is strictly convex (with ), ensuring a unique global minimum.
Modeling log-odds and minimizing L2-regularized binary cross-entropy leverages to cancel denominator derivatives cleanly:
The resulting objective is strictly convex (with ), ensuring a unique global minimum.
🔄 Tensor Dimensions & Shape Flow
X (N, D) -> z = Xw+b -> sigmoid p -> grad_w = (1/N)X^T(p-y) + lambda*w -> update w, b
🛡️ Numerical Stability & Pitfalls Checklist
- Clip logits safely before exp/sigmoid and compute log-loss stably to prevent underflow/overflow.
- Exclude the bias term
bfrom L2 weight regularization so baseline class priors remain unbiased.
💻 Reference Code Implementation
import numpy as np
class LogisticRegressionL2:
def __init__(self, lr: float = 0.1, lambda_reg: float = 0.01, iters: int = 300):
self.lr = lr
self.lambda_reg = lambda_reg
self.iters = iters
self.w = None
self.b = 0.0
def fit(self, X: np.ndarray, y: np.ndarray):
N, D = X.shape
self.w = np.zeros(D)
self.b = 0.0
for _ in range(self.iters):
# 1. 线性预测与数值安全 Sigmoid
z = X @ self.w + self.b
z_clipped = np.clip(z, -88.0, 88.0)
p = 1.0 / (1.0 + np.exp(-z_clipped))
# 2. 计算误差项
error = p - y # (N,)
# 3. 梯度计算 (带 L2 权重衰减,偏置 b 通常不正则化)
grad_w = (1.0 / N) * (X.T @ error) + self.lambda_reg * self.w
grad_b = (1.0 / N) * np.sum(error)
# 4. 步进
self.w -= self.lr * grad_w
self.b -= self.lr * grad_b
def predict_proba(self, X: np.ndarray) -> np.ndarray:
z = np.clip(X @ self.w + self.b, -88.0, 88.0)
return 1.0 / (1.0 + np.exp(-z))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
X = np.array([[1.0], [2.0], [-1.0], [-2.0]])
y = np.array([1.0, 1.0, 0.0, 0.0])
clf = LogisticRegressionL2(lr=0.5, iters=500)
clf.fit(X, y)
assert clf.predict_proba(np.array([[3.0]])) > 0.8
assert clf.predict_proba(np.array([[-3.0]])) < 0.2
print("✓ 逻辑回归二分类自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why is MSE a poor loss choice for Logistic Regression compared to Cross-Entropy?
Composing Sigmoid with MSE yields a non-convex loss whose gradient contains ; when predictions are confidently wrong, Sigmoid saturation zeroes out the gradient. Cross-entropy cancels that factor cleanly, making gradients proportional to and restoring convexity.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.