A1087
Title: Sequential testing for recurring users
Authors: Zhimei Ren - University of Pennsylvania (United States)
Ruohan Zhan - Hong Kong University of Science and Technology (Hong Kong) [presenting]
Abstract: A/B testing is widely used by online platforms to evaluate interventions such as interface changes, algorithm updates, or pricing strategies. Traditional A/B tests require pre-specified sample sizes to control Type-I error using p-values. However, tech firms often monitor experiments continuously, leading to optional stopping and inflated Type-I error due to peeking and p-hacking. Sequential testing methods address this issue by enabling anytime-valid inference using e-processes, which ensure time-uniform Type-I error control. However, existing methods assume independent user observations and randomized treatment assignments. These assumptions do not hold in modern platforms where recurring users generate repeated observations, introducing dependence and reducing treatment randomization over time. This invalidates standard e-process constructions. A novel sequential testing framework is developed that accommodates recurring users while maintaining statistical validity. The approach is anytime-valid, nonparametric, and theoretically sound for dependent observations. In collaboration with a major tech firm, the method is implemented in their experimentation system, integrating sequential testing into real-world decision-making. The framework also supports adaptive experimentation and efficient multiple-hypothesis testing, improving experimentation practices on online platforms.
