The results below were produced by Wesolowsky's SCheck program.

 

Two of the 11 pairs below, {36, 70} and {95, 128}, were not flagged by Lertap's RSA.  The two programs agreed on the other 9 pairs.  (But be sure to read the text below the table -- the two programs agree exactly if Lertap's options are changed.)

 

   

 

When Lertap's RSA analysis was initially run, producing the results seen on the previous page, four items were omitted due to "overly-popular distractors", that is, distractors (incorrect answers) that were selected more frequently than the items' correct answers.  Excluding such items is recommended when Lertap RSA runs.  However, if the correct answer is selected almost as often as the "popular distractor", and if the item's discrimination is at least 0.20, then leaving the item in the analysis may well be appropriate.

 

In this case, if all four initially excluded items, I2, I12, I14, and I20, are included, then a Lertap RSA analysis will include 10 of the pairs identified by SCheck, missing out only the {95, 128} pair.  Here is the RSAcases report after the four items have been included:

 

 

Now, a look at the bottom of the corresponding RSAsig report:

 

 

The {S95, S128} pair of students has an H-H Index of 7.00, and a sigma greater than 5 (in absolute value, that is, ignoring the negative sign on -5.15).

 

If the EEIC cutoff is now changed to 6 (instead of 8), then the {S95, S128} pair will also be added to Lertap's RSAcases list, resulting in total agreement with SCheck:

 

 

Harpp & Hogan (1993) suggested an EEIC=6 setting but modified it to EEIC=8 in Harpp, Hogan, & Jennings (1996).  (Reference list is here).

 

The EEIC=8 setting can be viewed as conservative in that it makes it more difficult for a pair's results to be flagged as being possibly indicative of collusion.

 

This example has shown that looking at RSAsig results might suggest what could happen if the EEIC cutoff setting is changed: initially we had 10 student pairs in the RSAcases report when EEIC=8, but a look at the bottom of the corresponding RSAsig report signaled that we might want to set EEIC=6. When we did, the number of student pairs in RSAcases went up to 11.

 

It would bear repeating, perhaps, to again say that none of these results represent conclusive evidence of cheating.

 

---

 

Tidbit: the Windows version of Lertap 5 allows for some RSA settings to be changed on-the-fly, making it easy to see the effects of changing them.