A0938
Title: Researcher degrees of freedom and associated challenges in designing statistical parametric simulation studies
Authors: Felix Julian David Lange - LMU Munich (Germany) [presenting]
Abstract: Parametric simulation studies are indispensable for evaluating and comparing statistical methods. Designing such a study involves numerous decisions, and researchers generally have a great deal of flexibility when making decisions about the various design aspects, including parameter values, performance measures, and methods involved. While this flexibility, commonly referred to as researcher degrees of freedom, is part of the appeal of simulation studies, it also presents considerable challenges and risks. Not only is there a risk that the large number of decisions leads to arbitrary choices. The researcher-specified conditions might also be unrealistic, which is problematic because statistical simulation studies are frequently used as a basis for practical recommendations about methods. Researchers may also unintentionally or intentionally make biased choices that favor certain methods or outcomes, such as demonstrating that a particular method is superior. These issues, along with some potential remedies, are discussed. Additionally, it is illustrated how the researcher's degrees of freedom in the design can affect a study's results and lead to findings that do not generalize well.
