A0970
Title: Model diagnostics and forecast evaluation for quantiles
Authors: Tilmann Gneiting - Heidelberg Institute for Theoretical Studies (Germany)
Daniel Wolffram - Karlsruhe Institute of Technology (Germany)
Johannes Resin - Heidelberg University (Germany)
Kristof Kraus - Heidelberg Institute for Theoretical Studies (Germany)
Johannes Bracher - Karlsruhe Institute of Technology (Germany)
Timo Dimitriadis - Heidelberg University (Germany)
Veit Hagenmeyer - Karlsruhe Institute of Technology (Germany)
Alexander Jordan - HITS gGmbH, Heidelberg Institute for Theoretical Studies (Germany) [presenting]
Sebastian Lerch - Karlsruhe Institute of Technology (Germany)
Kaleb Phipps - Karlsruhe Institute of Technology (Germany)
Melanie Schienle - Karlsruhe Institute of Technology (Germany)
Abstract: Model diagnostics and forecast evaluation are closely related tasks, with the former concerning in-sample goodness (or lack) of fit and the latter addressing predictive performance out-of-sample. The ubiquitous setting is reviewed in which forecasts are cast in the form of quantiles or quantile-bounded prediction intervals. Unconditional calibration is distinguished, which corresponds to classical coverage criteria, from the stronger notion of conditional calibration, as can be visualized in quantile reliability diagrams. Consistent scoring functions - including, but not limited to, the widely used asymmetric piecewise linear score or pinball loss - provide for comparative assessment and ranking, and link to the coefficient of determination and skill scores. The use of these tools is illustrated in Engel's food expenditure data, the global energy forecasting competition 2014, and the US COVID-19 forecast hub.
