A0322
Title: Fast learning rates of averaged stochastic gradient descent for over-parameterized neural networks
Authors: Atsushi Nitanda - Kyushu Institute of Technology (Japan) [presenting]
Taiji Suzuki - University of Tokyo / RIKEN-AIP (Japan)
Abstract: The convergence of averaged stochastic gradient descent for over-parameterized two-layer neural networks on the regression problem is analyzed. We consider a condition where the target function is contained in the reproducing kernel Hilbert space spanned by the neural tangent kernel, and the network width is sufficiently large such that the learning dynamics fall into the neural tangent kernel regime. Under this setting, we show the global convergence of the averaged stochastic gradient descent and derive the fast convergence rate by exploiting the complexities of the target function and the neural tangent kernel depending on the data distribution.
