A1178
Title: Algorithm-dependent learning analysis of over-parameteried neural networks
Authors: Yuan Cao - The University of Hong Kong (Hong Kong) [presenting]
Junwei Lu - Harvard T.H. Chan School of Public Health (United States)
Abstract: Modern machine learning models, such as large language models, often contain a huge number of parameters. For such over-parameterized models, there can be infinitely many minimizers of the training loss function, and different training algorithms may thus converge to different solutions. While these solutions may all yield zero training error, they can have drastically different prediction errors on new data. Therefore, understanding modern machine learning requires analyzing the impact of training algorithms on generalization performance. Recent studies on algorithm-dependent learning guarantees for over-parameterized neural networks are discussed. Theoretical analyses of the benign overfitting phenomenon are first presented, and then the test performance of gradient descent, full-batch Adam, and stochastic Adam are compared. The findings highlight the crucial role of optimization algorithms in determining the practical success of modern large-scale neural networks.
