ResNet Skip Connections

A residual block computes $y = F(x, \{W_i\}) + x$ — the layer learns the residual $F(x) = \mathcal{H}(x) - x$ instead of the direct map $\mathcal{H}(x)$. In backprop, $\frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}$: the identity shortcut gives a gradient highway along which the gradient can flow through any number of layers via the constant 1 term, mitigating vanishing gradients (equivalent to ensembling many shallow networks). Degradation: a 56-layer plain network has higher training error than a 20-layer one, and not due to overfitting — the identity map is simply hard to learn; residuals rewrite the goal as 'learn F→0', so deep networks are at least no worse than shallow ones, which is why ResNet-152 trains stably.

a standard component of nearly all deep CNNs/Transformers (same idea as Pre-Norm); interview staples — what is the degradation problem, why do residuals make thousand-layer networks trainable, why must the shortcut be identity.

optimization degradation and vanishing gradients beyond a depth threshold — with identity shortcuts, accuracy improves monotonically with depth without a hard ceiling; ImageNet top-5 error drops from ~7.3% (VGG) to ~4.6% (ResNet-152), and 152-layer models became the default backbone for vision.

Derive the residual-block gradient $\frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}$ and explain why the identity shortcut mitigates vanishing gradients.

What is the degradation problem? Why does a 56-layer net have higher training error than a 20-layer one without overfitting?

Why must the shortcut be the identity $x$ rather than a learned transform such as 1×1 conv?

When F changes channels/resolution, how do you match the shortcut dimensions, and what does each option cost?

How does ResNet make 152-layer (even 1000-layer) training possible? Answer from both optimization and gradient perspectives.