A0412
Title: Examining the usage of rating scale in subjective creativity assessments through partial credit model
Authors: Sujie Yang - University of Science and Technology of China (China) [presenting]
Jue Wang - University of Science and Technology of China (China)
Abstract: The evaluation of subjective creativity assessments is challenging due to the involvement of raters and a lack of scoring criteria. Raters' cognitive process is viewed as a black box. A psychometric framework is introduced for examining subjective creativity assessments along with a set of guidelines for evaluating the usage of rating scales. Within this framework, the focus is on the selection of an appropriate measurement model for analyzing ratings that reflect quite different judgment decisions among raters. In particular, the use of a modified partial-credit Rasch model is proposed with a rater facet that allows for the estimation of a unique threshold structure for each rater. This model can also be viewed as a reduced version of the many-facet Rasch model that provides more information regarding the unique category usages by raters and examination of various rater effects. An empirical data analysis is conducted using rating scores obtained from a subjective creativity assessment based on science tasks. Results indicate very different usages of rating scales among raters and reveal certain effects on individual raters. Discussions on the uses of the partial credit model and implications for improving the rating procedure within the context of subjective creativity assessments are provided.
