Title: The state of industry statistics in online AB experiments
Authors: Michael Lindon - Optimizely (United States) [presenting]
Abstract: A survey of the most popular statistical methodologies used in the experimentation and growth industry concerning online A/B experiments is provided. The online experimentation space concerns running numerous web based experiments on millions of users to measure the effect of various treatments in increasing metrics of interest - user signups, revenue spend per user and user conversions for example. Unfortunately statistical best practices are often overlooked in this domain, for which the usual offenders are incorrectly adjusting for multiple testing and continuous monitoring. The literature for sequential hypothesis testing is reviewed, its application to online experimentation, and some very recent developments in extending these methodologies to combinations of composite hypotheses are discussed. A juxtaposition of Bayesian, frequentist and conditional frequentist approaches to this problem are presented throughout, with a discussion of their commonalities and differences.