Title: Ensemble of regularized linear models
Authors: Ruben Zamar - University of British Columbia (Canada)
Stefan Van Aelst - University of Leuven (Belgium) [presenting]
Abstract: A new approach is presented for building ensembles of regularized linear models. The approach consists in optimizing an objective function that encourages sparsity within each model and diversity among the models. The procedure works on top of a given penalized linear regression estimator (e.g., Lasso, Elastic Net, SCAD) by fitting the given estimator to possibly overlapping subsets of features, while at the same time encouraging diversity among the subsets, to reduce the correlation between the predictions from each fitted model. The predictions from the models are then aggregated. For the case of an Elastic Net penalty and orthogonal predictors, we give a closed form solution for the regression coefficients in each of the ensembled models. We prove the consistency of our method in possibly high-dimensional linear models, where the number of predictors can increase with the sample size. An extensive simulation study and real-data applications show that the proposed method systematically improves the prediction accuracy of the base linear estimators being ensembled. Possible extensions to GLMs and other models are discussed.