B0438
Title: Quantifying discovery in astrophysics and particle physics
Authors: David van Dyk - Imperial College London (United Kingdom) [presenting]
Sara Algeri - Imperial College London (United Kingdom)
Jan Conrad - University of Stockholm and Imperial College London (Sweden)
David Jones - Texas A&M University (United States)
Abstract: The question of how best to compare models and select among them has dogged statisticians for decades. The difficulties with interpreting $p$-values and the strong dependence of Bayes Factors on the choice of prior distribution are well known. We explore a class of non-standard model comparison problems that are important in astrophysics and in high-energy physics. The search for the Higgs boson, for example, involved quantifying evidence for a narrow component added to a diffuse background distribution. The added component corresponds to the Higgs mass distribution and cannot be negative. Thus, not only is the null distribution on the boundary of the parameter space, but the added components location is unidentifiable under the null. Because many researchers have a strong preference for frequency-based statistical methods, they employ a sequence of likelihood ratio tests on a grid of possible null values of the unidentifiable location parameter. We compare this with a Bayesian strategy that employs a prior distribution on the location parameter and show how this prior automatically corrects for the multiple testing inherent in the standard method. The Bayesian procedure is significantly more conservative in that it avoids the well-known tilt of p-values toward the alternative when testing a precise null hypothesis. Finally, we discuss the circumstance under which the dependence of the Bayes Factor can be interpreted as a natural correction for multiple testing.
Journal