A0181
Title: Model class selection
Authors: Ryan Cecil - University of Pittsburgh (United States) [presenting]
Lucas Mentch - University of Pittsburgh (United States)
Abstract: Classical model selection generally seeks to find a single model within a particular class that optimizes some pre-specified criteria, such as maximizing a likelihood or minimizing a risk. More recently, there has been an increased research focus on model set selection (MSS), where the aim is to identify a subset of all models such that no model in the selected subset is significantly worse than the empirically optimal one. This subset of optimal models is sometimes referred to as the Rashomon set. The MSS framework is generalized one step further by introducing the idea of model class selection (MCS). In MCS, multiple model collections or classes are assumed, and all such collections that do (or do not) contain at least one optimal model are sought for identification. As a direct consequence, for particular datasets we are able to investigate formally whether classes of simpler and more interpretable statistical models are able to perform on par with more complex black-box machine learning models. In other words, as it has become relatively common for practitioners to rule out classical models a priori because the data is assumed to be too large and/or complex, another means of evaluating whether (and when) such actions are justified is proposed. A variety of simulated and real-data experiments are provided.
