B1584
Title: Safe testing \& Probabilities: A unified treatment of optional stopping, untrustworthy priors and misspecification
Authors: Peter Grunwald - CWI and Leiden University (Netherlands) [presenting]
Abstract: P-values give rise to valid Type-I error probabilities if sampling plan and significance level are chosen independently of the observed data, and give un-interpretable numbers in other situations such as with optional stopping. We introduce S-values, an alternative to p-values which remain valid, under specific loss functions, if the significance level may depend on the data. We review test martingales, which even remain valid under optional stopping. We introduce `safety' as an analogue for `validity' for methods such as Bayes and fiducial that output a distribution. This leads to a general calculus of `validity' and `safety' that should give us a much better idea of what possibly misspecified models can be used for and what not. Some examples: generalized Bayesian inference with a (high-dimensional) linear regression model, even if severely misspecified, is safe for squared error prediction and for assessing the quality of those predictions; it is unsafe for just about any other loss function. The fiducial posterior is safe for making confidence statements when the sampling plan is fixed in advance, but not under optional stopping. Bayesian null hypothesis testing with point null is safe for Type 0 Error Probabilities, irrespective of the prior, even under optional stopping; with composite hypothesis testing it is not safe for Type 0 Error, not even with a fixed stopping time.
