CRoNoS & MDA 2019: Start Registration
View Submission - CRONOSMDA2019
A0249
Title: Distributional properties of Bayesian deep neural networks Authors:  Julyan Arbel - Inria (France) [presenting]
Abstract: Deep Bayesian neural networks with Gaussian priors on the weights and ReLU-like nonlinearities are investigated, shedding light on novel regularization mechanisms at the level of the units of the network, both pre- and post-nonlinearities. The main thrust is to establish that the units prior distribution becomes increasingly heavy-tailed with depth. We show that first layer units are Gaussian, second layer units are sub-Exponential, and we introduce sub-Weibull distributions to characterize the deeper layers units. Bayesian neural networks with Gaussian priors are well known to induce the weight decay penalty on the weights. In contrast, our result indicates a more elaborate regularization scheme at the level of the units. This result provides new theoretical insight on deep Bayesian neural networks, underpinning their natural shrinkage properties and practical potential.