A1800
Title: A snapshot of influence: A local data attribution framework for online reinforcement learning
Authors: Jiaqi Ma - University of Illinois Urbana-Champaign (United States) [presenting]
Abstract: Online reinforcement learning (RL) excels in complex, safety-critical domains but suffers from sample inefficiency, training instability, and limited interpretability. Data attribution provides a principled way to trace model behavior back to training samples, yet existing methods assume fixed datasets, which is violated in online RL where each experience both updates the Policy and shapes future data collection. The aim is to investigate data attribution for online RL, focusing on the widely used Proximal Policy Optimization (PPO) algorithm. A local attribution framework is established, interpreting model checkpoints with respect to records in the recent training buffer. Two target functions are designed, capturing agent action and cumulative return respectively, and each record's contribution is measured through gradient similarity between its training loss and these targets. The framework is demonstrated through three concrete applications: diagnosis of learning, temporal analysis of behavior formation, and targeted intervention during training. Leveraging this framework, an algorithm called iterative influence-based filtering (IIF) is further proposed for online RL training that iteratively performs experience filtering to refine Policy updates.
