Stacking the Deck

For tasks like image or speech recognition, why do deep networks tend to represent complex functions far more efficiently than shallow-but-wide networks with the same total parameter count?

Each added layer increases model capacity by the same fixed amount regardless of how layers compose

Each layer composes the simpler features learned by earlier layers into increasingly abstract, hierarchical representations, needing exponentially fewer units than a shallow net to express the same function class

Adding depth makes the loss surface convex, eliminating local minima

Deeper networks always converge faster because gradients propagate more reliably through more layers