A0797
Title: Distance-based reliability
Authors: Philip Reiss - University of Haifa (Israel) [presenting]
Meng Xu - University of Haifa (Israel)
Ivor Cribben - Alberta School of Business (Canada)
Abstract: The intraclass correlation coefficient (ICC) is a classical index of measurement reliability. With the advent of new and complex types of data for which the ICC is not defined, there is a need for new ways to assess reliability. To meet this need, a distance-based ICC (dbICC) defined in terms of arbitrary distances among observations is proposed. It is shown that naive bootstrap confidence intervals for the dbICC suffer from undercoverage, and a bias correction is introduced to remedy this. The Spearman-Brown (SB) formula, which shows how more intensive measurement increases reliability, is extended to encompass the dbICC. The generalized SB formula depends on a notion of measurement intensity that generalizes simple averaging over multiple measurements. The dbICC is illustrated by analyzing test-retest reliability in several settings, including brain connectivity matrices derived from functional magnetic resonance imaging, as well as complex phenotypes derived from experience sampling and psychotherapy research.
