A0645
Title: Detecting interference in A/B testing with increasing allocation
Authors: Kevin Han - Meta (United States)
Shuangning Li - University of Chicago (United States) [presenting]
Jialiang Mao - LinkedIn Corp (United States)
Han Wu - Two Sigma (United States)
Abstract: In the past decade, the technology industry has adopted A/B testing to guide product development and make business decisions. A/B tests are often implemented with increasing treatment allocation: the new treatment is gradually released to an increasing number of units through a sequence of randomized experiments. In scenarios such as experimenting in a social network setting or in a bipartite online marketplace, interference among units may exist, harming the validity of simple inference procedures. A procedure is introduced to test for interference in A/B testing with increasing allocation. The procedure can be implemented on an existing A/B testing platform with a separate flow and does not require a specific interference mechanism. In particular, two permutation tests that are valid under different assumptions are introduced. Firstly, a general statistical test is introduced for interference, requiring no additional assumption. Secondly, a testing procedure is introduced that is valid under a time-fixed effect assumption. The testing procedure has very low computational complexity, is powerful, and formalizes a heuristic algorithm already implemented in the industry. The performance of the proposed testing procedure is demonstrated through simulations on synthetic data. Finally, one application of the proposed methods at LinkedIn is discussed, where a screening step is implemented to detect potential interference in marketplace experiments.
