A1792
Title: Estimating treatment effects in finite horizon experiments with rolling enrollment
Authors: Jaume Vives-i-Bastida - Stanford GSB (United States) [presenting]
Stefan Wager - Stanford University (United States)
Ramesh Johari - Stanford (United States)
Abstract: All experiments unfold over a finite horizon. In combination with rolling enrollment of experimental subjects, this complicates the estimation of treatment effects. Subjects arriving later in the experiment have shorter observation horizons, leading to a Censoring problem. Standard methods relying on simple difference-in-means (DiM) estimators conflate early subject arrivals with treatment effects, leading to potentially misleading long-term estimates. Formal modeling of the Censoring process inherent in rolling enrollment finite horizon experiments identifies the adoption/conversion rate curve -the Probability that a subject realizes its outcome within a fixed time- as the primary object of interest. To estimate the adoption rate curve, an Inverse Probability of Censoring Weighting estimator (IPCW) and an Augmented Inverse Probability of Censoring Weighting (AIPCW) estimator are introduced. Theoretical guarantees are provided for the IPCW and AIPCW estimators, and the efficiency gains of the AIPCW estimator are quantified theoretically. A framework for targeting long-run effects is considered, and the MSE optimal estimators among a class of adoption curve weighting estimators are derived. Extensive simulations and applications to A/B tests run at Airbnb highlight the differences between alternative estimators and demonstrate the importance of estimating the conversion rate curve directly for evaluating treatment effects.
