M3-008M3: Deep Learning FoundationsWeight InitializationEasy
⚡Flashcard Drill➔Mastery:
Why All-Zero Weight Initialization Fails in Neural Networks
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
All-zero initialization causes all hidden neurons within the same layer to compute identical activations and receive identical gradients, preventing symmetry breaking.
📌 Key Takeaways
- •Symmetry preservation: if , every neuron in layer computes , producing identical outputs
- •Identical gradients: backpropagation calculates identical partial derivatives for all parallel weights
- •Capacity collapse: no matter how long the network is trained, the entire layer behaves as a single individual neuron
📐 Mathematical Derivations
Proof of Symmetry Collapse: Let layer have weight matrix with .
- Forward pass: For any input vector : . The activations are identical for all .
- Backward pass: Downstream error delta . If the next layer weights are also symmetric, is identical for all . The gradient with respect to weights is: .
Because for all , the gradient is identical across all rows: .
- Gradient descent update: . All rows remain strictly identical for all future iterations. The network is mathematically constrained to a rank-1 subspace.
- Forward pass: For any input vector : . The activations are identical for all .
- Backward pass: Downstream error delta . If the next layer weights are also symmetric, is identical for all . The gradient with respect to weights is: .
Because for all , the gradient is identical across all rows: .
- Gradient descent update: . All rows remain strictly identical for all future iterations. The network is mathematically constrained to a rank-1 subspace.
🏭 Production Trade-offs
Biases exception: While weights must be randomized to break symmetry, biases can safely be initialized to zero () because distinct random weights ensure neurons already compute divergent activations.
⚠️ Common Interview Pitfalls
- ✕Believing that non-linear activation functions will break symmetry; non-linearities preserve identical activations when inputs are identical
- ✕Assuming bias vectors must also be initialized randomly; randomizing biases can destabilize initial variance without providing benefit
🎯 Interviewer Follow-up Inquiries
- ?Can linear regression or logistic regression be initialized with all zeros? Why is it safe there?
- ?What specific architectural components (like residual connections) can benefit from selective zero initialization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚