A0855
Title: Optimal design for A/B testing in time series experiments
Authors: Chengchun Shi - LSE (United Kingdom) [presenting]
Abstract: Time series experiments, in which experimental units receive a sequence of treatments over time, are frequently employed in many technological companies to evaluate the performance of a newly developed policy, product, or treatment relative to a baseline control. Many existing A/B testing solutions assume a fully observable experimental environment that satisfies the Markov condition, which often does not hold in practice. The optimal design for A/B testing is studied in partially observable environments. A controlled (vector) autoregressive moving average model is introduced to capture partial observability. A small signal asymptotic framework is introduced to simplify the analysis of asymptotic mean squared errors of average treatment effect estimators under various designs. Two algorithms are developed to estimate the optimal design: one utilizing constrained optimization and the other employing reinforcement learning. The superior performance of the designs is demonstrated using a dispatch simulator and two real datasets from a ride-sharing company.
