B1173
Title: Instability of deep learning models in industrial NLP problems
Authors: Helen Xie - Apple (United States) [presenting]
Abstract: While many industrial companies turn to deep learning and/or machine learning models to achieve smart solutions for complicated NLP problems, a stable model is desired but sometimes hard to reach. Such a problem might never show up in academia, but only industrial. Several main reasons can be the design of infrastructure, algorithm and GPU related issues, and inconsistency in human-labelled data. For industrial, infrastructure is usually tuned to be able to hold trillions of traffics smoothly and efficiently but not highly consistently. Implementation of the experimental level algorithm to production has to be re-shaped, and stability usually is not the focus. Updating to the newest GPU can be slow. Human labels can constantly be changing as the algorithm changes. As a result, several statistic methods are applied to restrict models with high fluctuations (CI 95\%) to be shipped. Unfortunately, even with such restrictions, instability within models still exists, but from 32.2\% to 23.4\% in accuracy, using simulated models and data.
