Feed
On the stability of learning
Learning can blow up when correct-on-average feedback arrives too rarely and too noisily.
2
Ratio s/ρ where learning destabilizes
Above this ratio, errors flip sign and grow every update.
How it works
Feedback is intermittent
Synapses in the brain may get gradient signals rarely, at random times, with noise, even if correct on average.
Model random arrivals
Gradients arrive by a Poisson process at rate ρ. Track weight variance in linear, two-layer, and deep linear networks.
Critical arrival rate
Below a threshold ρc, variance diverges while the average still converges. Regularization, pooling, and averaging lower ρc.
From the paper