Abstract(s)
The use of the aggregate scoring method for scoring concordance tests requires the weighting of test items
to be derived from the performance of a group of experts who take the test under the same conditions as
the examinees. However, the average score of experts constituting the reference panel remains a critical
issue in the use of these tests. This study aims to examine the distribution of panelists’ scores on the
judgment concordance test (JCT) using the aggregate scoring method. A test composed of 32 items was
developed and completed by 14 experts. The mean scores of the experts were calculated based on whether
their choices of response categories for the 32 JCT items were included or excluded. Descriptive statistics
were conducted. The mean scores of the experts showed a difference of 5.76%, depending on the
approach used. The approach that excludes the experts’ response category choices was found to be more
penalizing (76.16%±8.9) than the method including their own choices (81.92%±8.1). It is recommended
that researchers make their computational approaches explicit in addition to outlining the distribution of
expert results retained for the purpose of determining scores in the concordance tests. Further research is
required to refine our understanding of the quality of score-setting in this type of test.