Title: Natural gradient factor variational approximations with applications to deep neural network models
Authors: Minh-Ngoc Tran - University of Sydney (Australia) [presenting]
Robert Kohn - University of New South Wales (Australia)
David Nott - National University of Singapore (Singapore)
Nghia Nguyen - University of Sydney (Australia)
Abstract: Deep neural networks (DNNs) are a powerful tool for functional approximation. We describe flexible versions of generalized linear and generalized linear mixed models incorporating basis functions formed by a deep neural network. The consideration of neural networks with mixed effects seems little used in the literature, perhaps because of the computational challenges of incorporating subject specific parameters into already complex models. With this in mind, we suggest a computationally efficient variational inference method useful for such models but also applicable more generally. In particular, we develop a natural gradient Gaussian variational approximation method incorporating a factor structure for the covariance matrix which provides a scalable approach to approximate Bayesian inference, even in high dimensions. The use of the natural gradient allows faster and more stable convergence of the variational algorithm, and computation of the natural gradient can be achieved using fast conjugate gradient methods for iterative solution of linear systems, making use of the factor structure of the variational posterior covariance matrix. The proposed methods are illustrated in several examples for DNN random effects models and high-dimensional logistic regression with sparse signal shrinkage priors.