Examination of the aggregate scoring method in a judgment concordance test

Abstract

The use of the aggregate scoring method for scoring concordance tests requires the weighting of test items to be derived from the performance of a group of experts who take the test under the same conditions as the examinees. However, the average score of experts constituting the reference panel remains a critical issue in the use of these tests. This study aims to examine the distribution of panelists’ scores on the judgment concordance test (JCT) using the aggregate scoring method. A test composed of 32 items was developed and completed by 14 experts. The mean scores of the experts were calculated based on whether their choices of response categories for the 32 JCT items were included or excluded. Descriptive statistics were conducted. The mean scores of the experts showed a difference of 5.76%, depending on the approach used. The approach that excludes the experts’ response category choices was found to be more penalizing (76.16%±8.9) than the method including their own choices (81.92%±8.1). It is recommended that researchers make their computational approaches explicit in addition to outlining the distribution of expert results retained for the purpose of determining scores in the concordance tests. Further research is required to refine our understanding of the quality of score-setting in this type of test.