A1450
Title: Statistical analysis of interleaving designs in A/B tests
Authors: Weitao Cheng - HKUST (Hong Kong)
Ningyuan Chen - University of Toronto (Canada)
Pin Gao - CUHK Shenzhen (Hong Kong)
Nian Si - HKUST (Hong Kong) [presenting]
Abstract: A/B testing for assortment recommendation systems is studied, where each user is presented with a set of items and makes a choice according to a stochastic choice model with an outside option. The goal is to estimate the global treatment effect (GTE) between treatment and control assortments when only user behavior is observed. The standard user-level randomized experiment (UE), which is unbiased but exhibits high variance, is compared with interleaving-based designs that merge treatment and control assortments to reduce variance at the cost of bias. Focus is placed on power analysis and it is shown that both UE and interleaving-based designs control Type I error, while interleaving-based designs achieve higher statistical power. Optimal interleaving mechanisms under this tradeoff are further characterized. Moreover, the framework is extended to sequential settings, where users examine recommended items sequentially in ranked order from top to bottom. Simulation studies corroborate the theoretical findings and illustrate the conditions under which interleaving designs yield substantial efficiency gains.
