M2-094M2: Classical Machine LearningLogistic Regression & GLMMedium
⚡Flashcard Drill➔Mastery:
Multinomial Logistic Regression (Softmax Regression) and Parameter Identifiability
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Softmax regression generalizes logistic regression to classes; overparameterization requires fixing one class vector to zero (reference class) to ensure parameter identifiability.
📌 Key Takeaways
- •Softmax formulation:
- •Parameter redundancy: adding an arbitrary constant vector to all leaves probabilities completely unchanged
- •Identifiability fix: set , reducing free parameters from to
📐 Mathematical Derivations
Proof of Non-identifiability: Let .
If we replace each with for any vector :
.
Because infinitely many parameter sets produce identical likelihoods, the Fisher information matrix is singular. To achieve statistical identifiability, set reference class . Then for : .
In deep learning, all vectors are retained because L2 weight decay implicitly resolves non-identifiability by selecting the minimum-norm solution.
If we replace each with for any vector :
.
Because infinitely many parameter sets produce identical likelihoods, the Fisher information matrix is singular. To achieve statistical identifiability, set reference class . Then for : .
In deep learning, all vectors are retained because L2 weight decay implicitly resolves non-identifiability by selecting the minimum-norm solution.
🏭 Production Trade-offs
Engineering distinction: In classical statistics (e.g., R, statsmodels), setting is mandatory for parameter inference and p-values. In neural networks (PyTorch, TensorFlow), symmetric -vector parameterization with weight decay is standard because it simplifies GPU backpropagation.
⚠️ Common Interview Pitfalls
- ✕Attempting to invert the unregularized Fisher information matrix of a -class softmax without fixing a reference class
- ✕Interpreting softmax output probabilities as independent class likelihoods rather than constrained sum-to-one distributions
🎯 Interviewer Follow-up Inquiries
- ?How does L2 regularization implicitly select the unique minimum-norm parameterization in overparameterized softmax?
- ?What is the Independence of Irrelevant Alternatives (IIA) property in multinomial logit models?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚