A0408
Title: Random-walk debiased inference for contextual ranking model with application in large language model evaluation
Authors: Yichi Zhang - Statistics Department, Indiana University Bloomington (United States) [presenting]
Abstract: A debiased inference framework is proposed to infer the ranking structure in the contextual Bradley-Terry-Luce (BTL) model. For the pairwise item comparison, a novel random-walk debiased estimator is introduced to efficiently aggregate the estimating functions of different item pairs. To approximate the nuisance preference score functions in the debiased estimator, a nonparametric maximum likelihood-based method is further introduced that can leverage many loss-minimization methods, e.g., the spline regression and deep neural networks. With decently estimated nuisance functions, the debiased estimator yields a tractable distribution and achieves the semiparametric efficiency lower bound asymptotically. The method is further extended for multiple hypothesis testing by incorporating the multiplier bootstrap techniques and adapting it to accommodate the distributional shift of contextual variables. Thorough numerical studies are provided to validate the statistical properties of the method, and its applicability is showcased in evaluating large language models based on human preferences under different contexts.
