B0868
Title: The causal mediation analysis in the e-commerce industry
Authors: Xuan Yin - Etsy Inc (United States) [presenting]
Abstract: Causal mediation analysis is a formal statistical framework to reveal the underlying causal mechanism in randomized experiments. The analysis has been widely employed in various disciplines. However, it has not been applied to online A/B tests, the online randomized experiments in the daily practice of the internet industry. Perhaps it is because online A/B tests in the internet industry are primarily for evaluation: estimating and testing the average treatment effect. We will discuss two of our recent works on the development of causal mediation analysis for producing insights for search and recommendation systems in the e-commerce industry. (1) Based on some evidence, it is hypothesized that search and recommendation systems could compete for users' attention, which leads to degradation in the overall performance of the website. We utilize causal mediation analysis to verify the hypothesis and quantify the competition formally. (2) It is common in the internet industry to develop algorithms offline to power online products that contribute to business KPIs. Evaluation metrics of algorithms are usually different from business KPIs. It is not clear which evaluation metric, among all available ones, should be the north star to guide the development of algorithms in order to optimize business KPIs. We extend causal mediation analysis and develop a novel approach, which is easy to implement and to scale up, to pick the north star.
