L2Easy★ Core Essential · Industrial BedrockPart L · Classical ML & Statistical Simulation
Linear Regression: Closed-Form & Gradient Descent
Linear regression via closed-form normal equation `(X^T X)^-1 X^T y` and full-batch gradient descent.
⏱️ Time:
闭式解 O(N * D^2 + D^3);梯度下降 O(iters * N * D)💾 Space:
O(D^2)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def linear_regression_closed_form(X: np.ndarray, y: np.ndarray) -> np.ndarray:
"""正规方程解析解: w = (X^T X + eps*I)^-1 X^T y"""
# 添加偏置列 (Bias)
N = len(X)
X_b = np.hstack([np.ones((N, 1)), X])
# 加少量 L2 岭回归扰动保证矩阵严格可逆
XtX =
reg = 1e-6 * np.eye(XtX.shape[0])
return np.linalg.solve(XtX + reg, X_b.T @ y)
def linear_regression_gradient_descent(X: np.ndarray, y: np.ndarray, lr: float = 0.01, iters: int = 200) -> np.ndarray:
N, D = X.shape
X_b = np.hstack([np.ones((N, 1)), X])
w = np.zeros(D + 1)
for _ in range(iters):
grad =
w -= lr * grad
return w
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why does the normal equation fail when features D exceed samples N, and how is it fixed?
💡
Core Mental Anchor / Mnemonic
Solve X^T X w = X^T y for modest D, or step against residual X^T(Xw - y) for large D.
📐 Mathematical Derivation & Core Formula
### Least-Squares Normal Equation vs. Gradient Descent
Minimizing MSE gives gradient .
Setting the gradient to zero yields the normal equation when is invertible. For very large feature dimensions (where inversion is prohibitive), iterative gradient descent scales linearly per step.
Minimizing MSE gives gradient .
Setting the gradient to zero yields the normal equation when is invertible. For very large feature dimensions (where inversion is prohibitive), iterative gradient descent scales linearly per step.
🔄 Tensor Dimensions & Shape Flow
X (N, D) -> append bias column -> solve normal eq or iterate gradient descent -> w (D+1,)
🛡️ Numerical Stability & Pitfalls Checklist
- Prefer
np.linalg.solve(orlstsq) over explicitnp.linalg.inv(A) @ bfor speed and numerical conditioning. - Standardize heterogeneous feature scales when optimizing with gradient descent.
💻 Reference Code Implementation
import numpy as np
def linear_regression_closed_form(X: np.ndarray, y: np.ndarray) -> np.ndarray:
"""正规方程解析解: w = (X^T X + eps*I)^-1 X^T y"""
# 添加偏置列 (Bias)
N = len(X)
X_b = np.hstack([np.ones((N, 1)), X])
# 加少量 L2 岭回归扰动保证矩阵严格可逆
XtX = X_b.T @ X_b
reg = 1e-6 * np.eye(XtX.shape[0])
return np.linalg.solve(XtX + reg, X_b.T @ y)
def linear_regression_gradient_descent(X: np.ndarray, y: np.ndarray, lr: float = 0.01, iters: int = 200) -> np.ndarray:
N, D = X.shape
X_b = np.hstack([np.ones((N, 1)), X])
w = np.zeros(D + 1)
for _ in range(iters):
grad = (1.0 / N) * X_b.T @ (X_b @ w - y)
w -= lr * grad
return w
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
X = np.array([[1.0], [2.0], [3.0], [4.0]])
y = 2.0 * X[:, 0] + 1.0 # 理论: w0=1, w1=2
w_cf = linear_regression_closed_form(X, y)
assert np.isclose(w_cf[0], 1.0, atol=1e-3) and np.isclose(w_cf[1], 2.0, atol=1e-3)
print("✓ 线性回归闭式解自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why does the normal equation fail when features D exceed samples N, and how is it fixed?
When , , making singular and non-invertible. Adding L2 regularization (Ridge regression) replaces it with , guaranteeing strict positive definiteness and invertibility.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.