Scorer reliability of the KTSA

Abstract

PROBLEMHedlund and Mills(6) report scoring reliability results considerably at odds with those previously cited in work with the Kahn Test of Symbol Arrangement (KTSA). Craddick ( 2 ) has commented on the procedural and methodological techniques which explain many of these discrepancies. No previous study has utilized the most recent scoring principles. This study reports an investigation of KTSA scorer reliability based exclusively on the scoring principles developed by Hill and Latham(6s '). METHODOne hundred and eighty protocols were randomly selected from existing clinic files from the period January 1962 through February 1964. This sample represented a mean Patient Age of 30.3 years (SD = 10.6), and a mean Achieved Education of 12.4 years (SD = 2.3) with a mean of 10.4 months (SD = 7.2) between initial scoring and rescoring of responses. These were submitted to six trained technicians1 for independent scoring of responses in the subsections of (a) Reasons for Arrangements, (b) Symbolization, and (c) Reasons for Liking/Disliking. These results were then analyzed for intrascorer reliability using Spearman Rank Coefficients (rho). (lo-p. 2M)From the original group of 180 protocols, a random sample of 20 was drawn. Mean Age and Achieved Education of this sample was 33.1 years (SD = 11.0) and 12.2 years (SD = 1.9), respectively. Score-rescore time lapse for this sample waa 9.7 months (SD = 7.4). ttests for significance of mean differences in the two samples revealed no significant differences. Two experienced clinicians2 independently scored these 20 protocols for the same set of data as had been used in the intrascorer reliability analysis. The results of all eight scorers were analyzed for interscorer reliability utilizing phi coefficient of correlation (4). p. *l9) RESULTS~Under the null hypothesis of no agreement between a scorer's original and rescore decisions, intrascorer rho's for five of the scorers exceeded the .01 level of significance; the sixth scorer exceeded the .05 level. The null hypothesis was rejected. TABLE 1. PHI AND RHO COEFFICIENTS FOR 8 AND 6 SCORERS RESPECTI~LY scoring Median Phi Scorer Category Phi Range Number T h o .53 .91 .85 .32-.83 .83 -.98 .75 -.92 1 2 3 .59 .42-.75 4 .78 .60 -.87 5 .98 .95 .77* .87 .95 .88 . 7 1 -.97 6 .93 .77 .60 -.84 .79 .66 -.87 Median rho = .94 .84 . 7 5 -.96 *N = 9 Scoring categories. P .05 = ,600; P .01 = .783. 1A2C Alfred Byrne, SSgt John Cooper, A1C Louis Ramey, A2C Ralph Dukes, Jr., SSgt Alan Guerin, A1C Richard Iltis. Three scorers hold B.A. degrees, the remainder hold M.M. degreea All had at least 18 months' experience with the KTSA and at least 2 had worked with the test for 3 years. '1Lt. William R. Latham (M.Ed., 3 years' experience with the KTSA) and Lt. Colonel Theodore C. Kahn (Ph.D., test originator). 'Appreciation is hereb expressed to J. Frank Schmidt, Psychology Department, Trinity University for statistical cons&ation and advice.

Interscorer reliability and the KTSA.

“…This procedure loses subtle but important individual differences in interscorer comparisons. More recent re-evaluations (Clack, et al, 1966;Craddick and Stern, 1965;Hedlund and Mills, 1964) have attempted with varying success to correct some of the earlier methodological shortcomings. This paper represents the personal viewpoint of the author and is not to be construed as a statement of official Air Force policy.…”