A1146
Title: Selective risk control for AI model deployment with conformal e-values
Authors: Ying Jin - University of Pennsylvania (United States) [presenting]
Abstract: In deploying artificial intelligence (AI) models, selective prediction offers the option to abstain from making a prediction when uncertain about model quality. To fulfill its promise, it is crucial to enforce strict error control over cases where the model is trusted. Selective conformal risk control with e-values (SCoRE) is proposed as a new framework for deriving such decisions for any trained model. SCoRE offers two types of guarantees on the risk among "positive'' cases in which the system opts to trust and deploy the AI models. Built upon conformal inference and hypothesis testing ideas, the methods first construct a class of (generalized) e-values whose product with the unknown risk has an expectation no greater than 1. Such a property holds as long as the data are exchangeable without requiring any modeling assumptions. These e-values serve as inputs to hypothesis testing procedures to yield the binary decisions with desired finite-sample error control. The methods avoid the need for uniform concentration and can be readily extended to settings with distribution shifts and online streaming data. The proposed methods are evaluated with simulations and demonstrate their efficacy through applications to error management in AI-driven drug discovery, health risk prediction, and large language models.
