M2-094M2: Classical Machine LearningLogistic Regression & GLMMedium
⚡Flashcard Drill➔Mastery:

Multinomial Logistic Regression (Softmax Regression) and Parameter Identifiability

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-095 →
📐 Mathematical Definition
P(y=k∣x)=ewk⊤x∑j=1Kewj⊤xP(y=k\mid x)=\frac{e^{w_k^\top x}}{\sum_{j=1}^{K}e^{w_j^\top x}}
⚡ One-Sentence Core Answer
Softmax regression generalizes logistic regression to KK classes; overparameterization requires fixing one class vector to zero (reference class) to ensure parameter identifiability.

📌 Key Takeaways

  • •
    Softmax formulation: P(y=k∣x)=exp⁡(wkTx)∑j=1Kexp⁡(wjTx)P(y = k \mid x) = \frac{\exp(w_k^T x)}{\sum_{j=1}^K \exp(w_j^T x)}
  • •
    Parameter redundancy: adding an arbitrary constant vector cc to all wkw_k leaves probabilities completely unchanged
  • •
    Identifiability fix: set wK=0w_K = 0, reducing free parameters from K×dK \times d to (K−1)×d(K-1) \times d

📐 Mathematical Derivations

Proof of Non-identifiability: Let P(y=k∣x)=exp⁡(wkTx)∑j=1Kexp⁡(wjTx)P(y = k | x) = \frac{\exp(w_k^T x)}{\sum_{j=1}^K \exp(w_j^T x)}.
If we replace each wkw_k with wk′=wk+cw'_k = w_k + c for any vector c∈Rdc \in \mathbb{R}^d:
P′(y=k∣x)=exp⁡((wk+c)Tx)∑jexp⁡((wj+c)Tx)=exp⁡(wkTx)exp⁡(cTx)∑jexp⁡(wjTx)exp⁡(cTx)=P(y=k∣x)P'(y = k | x) = \frac{\exp((w_k + c)^T x)}{\sum_j \exp((w_j + c)^T x)} = \frac{\exp(w_k^T x) \exp(c^T x)}{\sum_j \exp(w_j^T x) \exp(c^T x)} = P(y = k | x).
Because infinitely many parameter sets produce identical likelihoods, the Fisher information matrix is singular. To achieve statistical identifiability, set reference class wK=0w_K = 0. Then for k<Kk < K: log⁡(P(y=k∣x)P(y=K∣x))=wkTx\log\left(\frac{P(y=k|x)}{P(y=K|x)}\right) = w_k^T x.
In deep learning, all KK vectors are retained because L2 weight decay λ∑k∥wk∥22\lambda \sum_k \|w_k\|_2^2 implicitly resolves non-identifiability by selecting the minimum-norm solution.

🏭 Production Trade-offs

Engineering distinction: In classical statistics (e.g., R, statsmodels), setting wK=0w_K=0 is mandatory for parameter inference and p-values. In neural networks (PyTorch, TensorFlow), symmetric KK-vector parameterization with weight decay is standard because it simplifies GPU backpropagation.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to invert the unregularized Fisher information matrix of a KK-class softmax without fixing a reference class
  • ✕
    Interpreting softmax output probabilities as independent class likelihoods rather than constrained sum-to-one distributions
🎯 Interviewer Follow-up Inquiries
  • ?
    How does L2 regularization implicitly select the unique minimum-norm parameterization in overparameterized softmax?
  • ?
    What is the Independence of Irrelevant Alternatives (IIA) property in multinomial logit models?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-093: Interaction and Polynomial Terms in Regression: Formulations and Regularization📋 Back to BankNext →M2-095: Relationship Between Logistic Regression and the Maximum Entropy Model