A1243
Title: A statistical framework for characterizing the response distribution of a large language model
Authors: William Wu - The University of Edinburgh (United Kingdom) [presenting]
Miguel de Carvalho - University of Edinburgh and Universidade de Aveiro (Portugal)
Abstract: Large language models (LLMs) often produce a spectrum of plausible responses to a single prompt; little attention has been paid to characterizing the conditional distribution of their responses, given a fixed query. The aim is to present a framework for analyzing LLM empirical distributions by leveraging sentence embeddings and similarity-based metrics. Total Recall and the RowBERT score are introduced as interpretable measures of variability of the ensemble responses and global similarity per reply. Building on these, a probabilistic scoring framework that quantifies the likelihood of a candidate response generated from a given ensemble is developed. Through simulations and experiments on multiple LLMs and open-source datasets, it is demonstrated that the empirical distribution formed by RowBERT score is stable, interpretable, and highly responsive to key generation parameters like temperature. The proposed heuristics-guided methodology offers a practical and interpretable framework for characterizing empirical output distributions from LLMs, providing critical insights into model robustness, trustworthiness, and the inherent variability of generated output.
