B1165
Title: Black box versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition
Authors: Vincent Dorie - New York University (United States)
Uri Shalit - New York University (United States)
Dan Cervone - New York University (United States)
Marc Scott - New York University (United States)
Jennifer Hill - New York University (United States) [presenting]
Abstract: Statisticians have made great strides towards assumption-free estimation of causal estimands in the past few decades. However this explosion in research has resulted in a breadth of inferential strategies that both create opportunities for more reliable inference as well as complicate the choices that an applied researcher has to make and defend. Relatedly, researchers advocating for new methods typically compare their method to (at best) 2 or 3 other causal inference strategies and test using simulations that may or may not be designed to equally tease out flaws in all the competing methods. The causal inference data analysis challenge, Is Your SATT Where Its At?, launched as part of the 2016 Atlantic Causal Inference Conference, sought to make progress with respect to both of these issues. The researchers creating the data testing grounds were distinct from the researchers submitting methods whose efficacy would be evaluated. Results from over 30 competitors in the two parallel versions of the competition (Black Box Algorithms and Do It Yourself Analyses) are presented along with post-hoc analyses that reveal information about the characteristics of causal inference strategies that yielded stronger inferences under a variety of circumstances. The most consistent conclusion was that the black box methods performed better overall than the user-controlled methods in all scenarios tested.
Journal