Title: Simulation-based optimal sequential Bayesian design using policy gradient reinforcement learning
Authors: Wanggang Shen - University of Michigan (United States)
Xun Huan - University of Michigan (United States) [presenting]
Abstract: Experiments are indispensable for learning and developing models in science and engineering. When experiments are expensive, a careful design of these limited data-acquisition opportunities can be immensely beneficial. Optimal experimental design, while leveraging the predictive capabilities of a simulation model, provides a rigorous framework to systematically quantify and maximize the value of an experiment. We focus on the design of a finite sequence of experiments, seeking design policies (strategies) that can (a) adapt to newly collected data during the sequence (i.e. feedback) and (b) anticipate future changes (i.e. lookahead). We cast this sequential learning problem in a Bayesian setting with information-based utilities, and solve it numerically via policy gradient methods from reinforcement learning. In particular, we directly parameterize the policies and value functions---thus adopting an actor-critic approach---and improve them using gradient estimates produced from simulated design sequences. The overall method is demonstrated on an algebraic benchmark and a sensor placement application for source inversion. The results provide intuitive insights on the benefits of feedback and lookahead, and indicate substantial computational advantages compared to previous numerical methods based on approximate dynamical programming.