Suppose we would like to compare two raters using a kappa statistic but the raters have different range of scores. This situation most often presents itself where one of the raters did not use the same range of scores as the other rater.
Let us consider an example where two graduate students where asked to rate 12 movies based on a scale from 1-3. One rater used all of the three scores possible while rating the movies whereas the other student did not like any of the movies and therefore rated all of them as either a 1 or a 2. Thus, the range of scores is the not the same for the two raters.
In Stata
Using the kap command in Stata it is no problem that there is an unequal range of scores for the two raters. The code to produce the kappa statistic would be:
kap rater1 rater2 Expected Agreement Agreement Kappa Std. Err. Z Prob>Z ----------------------------------------------------------------- 66.67% 33.33% 0.5000 0.1667 3.00 0.0013
In SAS
To obtain the kappa statistic in SAS we have to use the proc freq with the agree option in the tables statement. Unfortunately, SAS will only compute the statistics from the agree option if the two variables have exactly the same categories, which is not the case in this particular instance.
The solution to this problem is to create an extra observation, which will have the missing category for rater2. Thus, in this example we will create observation 13 which will be equal to 3 for both raters and then rater2 will have scores ranging from 1 to 3 just like rater1. Since we do not want this extra observation to influence the kappa statistic we now create a weight variable which will give a very small weight (i.e., weight = 0.001) to the extra observation and observations from the original data set are given comparatively large weights (i.e., weight = 1). The following code will give us the result that we want:
data kappa; set 'd:sasdatakappa'; proc print data=kappa ; run; Obs rater1 rater2 1 1 1 2 1 1 3 1 1 4 1 1 5 2 2 6 2 2 7 2 2 8 2 2 9 3 2 10 3 2 11 3 2 12 3 2 data kappa; if _n_=1 then do; rater1 =3; rater2 = 3; end; output; set kappa; run; data kappa; set kappa; if rater2 ne 3 then weight = 1; if rater2 = 3 then weight = 0.0001; run; proc print data=kappa; run; Obs rater1 rater2 weight 1 3 3 0.0001 2 1 1 1.0000 3 1 1 1.0000 4 1 1 1.0000 5 1 1 1.0000 6 2 2 1.0000 7 2 2 1.0000 8 2 2 1.0000 9 2 2 1.0000 10 3 2 1.0000 11 3 2 1.0000 12 3 2 1.0000 13 3 2 1.0000 proc freq data= kappa; tables rater1*rater2; test kappa; weight weight; run; Statistics for Table of rater1 by rater2 Simple Kappa Coefficient -------------------------------- Kappa 0.5000 ASE 0.1559 95% Lower Conf Limit 0.1944 95% Upper Conf Limit 0.8056
In SPSS
In SPSS we encounter the same difficulty as we saw in SAS, where the crosstab command will not compute the kappa statistic when the raters do not have the same range of scores. For SPSS version 11.0 and lower there is no way to obtain the kappa statistic in this situation. For SPSS version 11.5 we can actually obtain the kappa statistic and the solution is also very similar to the solution for SAS in that we generate an extra observation, which will have the missing category for rater2. Thus, in this example we will create observation 13 which will be equal to 3 for both raters and then rater2 will have scores ranging from 1 to 3 just like rater1. Since we do not want this extra observation to influence the kappa statistic we now create a weight variable which will give a very small weight (i.e., weight = 0.001) to the extra observation and observations from the original data set are given comparatively large weights (i.e., weight = 1).
First go into the data editor and manually add the extra observations to rater1 and rater2. Then use the following code to generate the weight variable and compute the kappa statistic.
COMPUTE weight = 1. IF rater2 = 3 weight = 0.001. EXE. WEIGHT BY weight. CROSSTABS /TABLES=rater1 BY rater2 /STATISTIC=KAPPA.