Neural Networks · 2022 · Journal article

Anomalous diffusion dynamics of learning in deep neural networks

Guozhang Chen, Cheng Kevin Qu, Pulin Gong

Stochastic gradient descent navigates fractal-like deep network loss landscapes through anomalous superdiffusion, enabling escape from sharp local minima.

The scientific question

How does stochastic gradient descent escape sharp local minima in deep networks?

Deep network training optimizes highly non-convex loss landscapes, yet standard Brownian diffusion approximations fail to explain how SGD reliably finds wide minima.

Schematic of an optimization trajectory through a non-convex loss landscape.
Figure 1 from the preprint ↗

The core idea

Statistical Physics of Loss Navigation

Using energy-landscape methods from complex physical systems, we track weight trajectories and diffusion exponents during stochastic gradient descent across convolutional and residual architectures.

What we found

Fractal Landscapes Drive Superdiffusive Jumps

Interaction between SGD and fractal-like loss geometry produces early superdiffusive jumps that escape sharp local minima, followed by late-stage subdiffusion as weights settle.

Continue the question

Related reading

Explore our research

Cite this work

Guozhang Chen, Cheng Kevin Qu, Pulin Gong. Anomalous diffusion dynamics of learning in deep neural networks. Neural Networks, 2022. doi:10.1016/j.neunet.2022.01.019.