A0205
Title: Design-based confidence sequences: A general approach to risk mitigation in online experimentation (work with Netflix)
Authors: Dae Woong Ham - University of Michigan (United States) [presenting]
Abstract: Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. Beyond aiding managerial decision-making, experiments mitigate risk by limiting the proportion of customers exposed to innovations. Since many experiments are conducted sequentially over time, an emerging strategy to further derisk the process is to allow managers to peek at the results as new data becomes available and stop the test if the results are statistically significant. The class of statistical methods that allow managers to peek and still provide valid inferences are often called anytime valid since they maintain proper uniform type-1 error guarantees. Existing anytime-valid approaches are extended to accommodate the more complex yet standard settings in time series, switchback, and panel experiments. To achieve this, the design-based approach is leveraged to focus on assumption-light and managerial relevant finite-sample estimands defined on the study participants as a direct measure of the risks incurred by companies. As a special case, results also provide a robust method for achieving always-valid inference in A/B tests. A variance reduction technique is further provided, incorporating modeling assumptions and covariates.
