A0707
Title: Estimation, forecasting and anomaly detection for nonstationary streams using adaptive estimation
Authors: Henrique Helfer Hoeltgebaum - Securonix (United Kingdom) [presenting]
Niall Adams - Imperial College London and University of Bristol (United Kingdom)
Cristiano Fernandes - Pontifical Catholic University of Rio de Janeiro (Brazil)
Abstract: Streaming data provides substantial challenges for data analysis. From a computational standpoint, these challenges arise from constraints related to computer memory and processing speed. Statistically, the challenges relate to constructing procedures that can handle so-called concept drift -- the tendency of future data to have different underlying properties to current and historic data. The issue of handling structure, such as trend and periodicity, remains a difficult problem for streaming estimation. We propose RAC (Real-Time Adaptive Component), a penalized-regression modelling framework which satisfies the computational constraints of streaming data, and provides capability for dealing with concept drift. At the core of the estimation process are techniques from adaptive filtering. The RAC procedure adopts a specified basis to handle local structure, along with a LASSO-like penalty procedure to handle over-fitting. We enhance the RAC estimation procedure with a streaming anomaly detection capability. Experiments with simulated data suggest the procedure can be considered as a competitive tool for a variety of scenarios, and an illustration with real cyber-security data further demonstrates the promise of the method.
