L2Easy★ Core Essential · Industrial BedrockPart L · Classical ML & Statistical Simulation

Linear Regression: Closed-Form & Gradient Descent

⚡Flashcard Drill➔

Linear regression via closed-form normal equation `(X^T X)^-1 X^T y` and full-batch gradient descent.

⏱️ Time: 闭式解 O(N * D^2 + D^3);梯度下降 O(iters * N * D)
💾 Space: O(D^2)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def linear_regression_closed_form(X: np.ndarray, y: np.ndarray) -> np.ndarray:
"""正规方程解析解: w = (X^T X + eps*I)^-1 X^T y"""
# 添加偏置列 (Bias)
N = len(X)
X_b = np.hstack([np.ones((N, 1)), X])
# 加少量 L2 岭回归扰动保证矩阵严格可逆
XtX =
reg = 1e-6 * np.eye(XtX.shape[0])
return np.linalg.solve(XtX + reg, X_b.T @ y)
def linear_regression_gradient_descent(X: np.ndarray, y: np.ndarray, lr: float = 0.01, iters: int = 200) -> np.ndarray:
N, D = X.shape
X_b = np.hstack([np.ones((N, 1)), X])
w = np.zeros(D + 1)
for _ in range(iters):
grad =
w -= lr * grad
return w
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why does the normal equation fail when features D exceed samples N, and how is it fixed?
Next: Logistic Regression with L2 Regularization →
💡

Core Mental Anchor / Mnemonic

Solve X^T X w = X^T y for modest D, or step against residual X^T(Xw - y) for large D.

📐 Mathematical Derivation & Core Formula

w∗=(X⊤X)−1X⊤y,∇wL=1NX⊤(Xw−y)w^* = (X^\top X)^{-1} X^\top y, \quad \nabla_w \mathcal{L} = \frac{1}{N} X^\top (Xw - y)
### Least-Squares Normal Equation vs. Gradient Descent
Minimizing MSE J(w)=12N∥Xw−y∥22J(w) = \frac{1}{2N}\|Xw - y\|_2^2 gives gradient ∇wJ(w)=1NX⊤(Xw−y)\nabla_w J(w) = \frac{1}{N} X^\top (Xw - y).
Setting the gradient to zero yields the normal equation w∗=(X⊤X)−1X⊤yw^* = (X^\top X)^{-1} X^\top y when X⊤XX^\top X is invertible. For very large feature dimensions DD (where O(D3)O(D^3) inversion is prohibitive), iterative gradient descent w←w−η∇wJ(w)w \leftarrow w - \eta \nabla_w J(w) scales linearly per step.

🔄 Tensor Dimensions & Shape Flow

X (N, D) -> append bias column -> solve normal eq or iterate gradient descent -> w (D+1,)

🛡️ Numerical Stability & Pitfalls Checklist

  • Prefer np.linalg.solve (or lstsq) over explicit np.linalg.inv(A) @ b for speed and numerical conditioning.
  • Standardize heterogeneous feature scales when optimizing with gradient descent.

💻 Reference Code Implementation

import numpy as np

def linear_regression_closed_form(X: np.ndarray, y: np.ndarray) -> np.ndarray:
    """正规方程解析解: w = (X^T X + eps*I)^-1 X^T y"""
    # 添加偏置列 (Bias)
    N = len(X)
    X_b = np.hstack([np.ones((N, 1)), X])
    # 加少量 L2 岭回归扰动保证矩阵严格可逆
    XtX = X_b.T @ X_b
    reg = 1e-6 * np.eye(XtX.shape[0])
    return np.linalg.solve(XtX + reg, X_b.T @ y)

def linear_regression_gradient_descent(X: np.ndarray, y: np.ndarray, lr: float = 0.01, iters: int = 200) -> np.ndarray:
    N, D = X.shape
    X_b = np.hstack([np.ones((N, 1)), X])
    w = np.zeros(D + 1)
    for _ in range(iters):
        grad = (1.0 / N) * X_b.T @ (X_b @ w - y)
        w -= lr * grad
    return w

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
X = np.array([[1.0], [2.0], [3.0], [4.0]])
y = 2.0 * X[:, 0] + 1.0 # 理论: w0=1, w1=2
w_cf = linear_regression_closed_form(X, y)
assert np.isclose(w_cf[0], 1.0, atol=1e-3) and np.isclose(w_cf[1], 2.0, atol=1e-3)
print("✓ 线性回归闭式解自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why does the normal equation fail when features D exceed samples N, and how is it fixed?
When D>ND > N, rank(X⊤X)≤N<D\mathrm{rank}(X^\top X) \le N < D, making X⊤XX^\top X singular and non-invertible. Adding L2 regularization (Ridge regression) replaces it with X⊤X+λIX^\top X + \lambda I, guaranteeing strict positive definiteness and invertibility.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: K-Means Clustering from ScratchBack to Part L & All 105Next: Logistic Regression with L2 Regularization →