M3-008M3: Deep Learning FoundationsWeight InitializationEasy
⚡Flashcard Drill➔Mastery:

Why All-Zero Weight Initialization Fails in Neural Networks

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-009 →
📐 Mathematical Definition
W=0⇒∇Wi=∇WjW=0\Rightarrow \nabla_{W_i}=\nabla_{W_j}
⚡ One-Sentence Core Answer
All-zero initialization causes all hidden neurons within the same layer to compute identical activations and receive identical gradients, preventing symmetry breaking.

📌 Key Takeaways

  • •
    Symmetry preservation: if W=0W = 0, every neuron in layer ll computes zj=0+bjz_j = 0 + b_j, producing identical outputs
  • •
    Identical gradients: backpropagation calculates identical partial derivatives ∂L∂wj\frac{\partial \mathcal{L}}{\partial w_j} for all parallel weights
  • •
    Capacity collapse: no matter how long the network is trained, the entire layer behaves as a single individual neuron

📐 Mathematical Derivations

Proof of Symmetry Collapse: Let layer ll have weight matrix W∈Rdout×dinW \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}} with Wjk=0,bj=0W_{jk} = 0, b_j = 0.
- Forward pass: For any input vector xx: zj=∑kWjkxk+bj=0z_j = \sum_k W_{jk} x_k + b_j = 0. The activations aj=g(zj)=g(0)a_j = g(z_j) = g(0) are identical for all j∈{1,…,dout}j \in \{1, \dots, d_{\text{out}}\}.
- Backward pass: Downstream error delta δj=∂L∂zj\delta_j = \frac{\partial \mathcal{L}}{\partial z_j}. If the next layer weights are also symmetric, δj\delta_j is identical for all jj. The gradient with respect to weights is: ∂L∂Wjk=δjxk\frac{\partial \mathcal{L}}{\partial W_{jk}} = \delta_j x_k.
Because δj=δ\delta_j = \delta for all jj, the gradient is identical across all rows: ∂L∂W1k=∂L∂W2k=⋯=∂L∂Wdoutk\frac{\partial \mathcal{L}}{\partial W_{1k}} = \frac{\partial \mathcal{L}}{\partial W_{2k}} = \dots = \frac{\partial \mathcal{L}}{\partial W_{d_{\text{out}}k}}.
- Gradient descent update: Wjk(t+1)=Wjk(t)−η∂L∂WjkW_{jk}^{(t+1)} = W_{jk}^{(t)} - \eta \frac{\partial \mathcal{L}}{\partial W_{jk}}. All rows remain strictly identical for all future iterations. The network is mathematically constrained to a rank-1 subspace.

🏭 Production Trade-offs

Biases exception: While weights WW must be randomized to break symmetry, biases bb can safely be initialized to zero (b=0b=0) because distinct random weights ensure neurons already compute divergent activations.
⚠️ Common Interview Pitfalls
  • ✕
    Believing that non-linear activation functions will break symmetry; non-linearities preserve identical activations when inputs are identical
  • ✕
    Assuming bias vectors must also be initialized randomly; randomizing biases can destabilize initial variance without providing benefit
🎯 Interviewer Follow-up Inquiries
  • ?
    Can linear regression or logistic regression be initialized with all zeros? Why is it safe there?
  • ?
    What specific architectural components (like residual connections) can benefit from selective zero initialization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-007: Gradient Checking: Principles, Finite Differences, and Implementation Details📋 Back to BankNext →M3-009: Xavier and Kaiming Initialization: Mathematical Variances and Suitable Activations