Title: A rank graduation measure to assess predictive accuracy
Authors: Emanuela Raffinetti - University of Milan (Italy) [presenting]
Paolo Giudici - University of Pavia (Italy)
Abstract: A very key point in the application of statistical and machine learning methods in Artificial Intelligence (AI) is the evaluation of their predictive accuracy. This is because the ``automatic'' choice of an action crucially depends on the predictive scenario under which that action will be implemented. Machine learning and statistics have provided, over the years, a number of summary measures aimed at measuring predictive accuracy, such as the root mean squared error, and the area under the ROC curve. Note that most of them are response-specific, and none of them can be applied to all types of response. This can be a problem in a complex situation, with different types of responses and, more generally, for an Artificial Intelligence system whose evaluation criteria should be determined exogenously and not endogenously. The aim is to present a more general measure which can improve predictive accuracy assessment in highly complex situations. More precisely, the proposed measure, called Rank Graduation index, is based on the comparison between the observed and the predicted response variable ranks, as in ordinal response models, but using, rather than the ranks themselves, the actual values of the response variable corresponding to both ranks, as in continuous or 0/1 response models. In order to appreciate the RG features, an application to credit scoring is also considered.