CMStatistics 2015: Start Registration
View Submission - CMStatistics
B1782
Title: Category prediction on short-answer question using dynamic scoring algorithm Authors:  Tomoya Okubo - The National Center for University Entarance Examinations (Japan) [presenting]
Abstract: We show some results on evaluating a dynamic scoring algorithm that enables us to score short-answer questions efficiently. The scoring system employs natural language processing techniques in order to calibrate similarity among examinees short-answers. The scoring system is able to show raters similarity index among the short-answers and it sorts the responses based on the similarity index; therefore, raters are able to score some responses at once. Further, the system dynamically predicts probabilities for each score-category based on fixed scores given by the raters. The information of prediction also helps the raters in the rating procedures. In high-stakes tests, stakeholders do not allow us to use full-automated scoring system because it cannot be always perfect. The scoring algorithm shown can be used in high-stakes tests since it is not a full-automated system and raters give rating for all the short-answers. It is important to have accountability for test-takers in high-stake testing. The scoring algorithm is able to keep accountability for ratings but also improves efficiency of scoring.