(Special topic inserted by Larry Nelson, 22 April 2016.)

 

Work was undertaken in April 2016 which resulted in another check on the working of Lertap 5's application of Harpp-Hogan methods.  In particular, I wanted to compare three cheat-checking programs: SCheck, Lertap 5, and Iteman 4. My results may be seen in Appendix A of this paper.

 

Once again I benefited from comments provided by George Wesolowsky, author of SCheck. I consider his work to be the gold standard in cheating detection, and he kindly assisted with the findings reported in the paper.

 

He reminded me of this correspondence which David Harpp sent to me in 2005 regarding the application of Harpp-Hogan methods:

 

Private communication from Professor David Harpp

to Larry Nelson, Curtin University, December 2005:

 

Analyses of well over 1000 examinations during

the past 13 year have shown that when this

number (EEIC/D) is ~1.5 or higher, at the same time

with a probability deviating from the norm by ~5

sigmas or higher there is a powerful indication

of cheating.  In virtually all cases to date where

the exam has ~30 or more questions as the lowest

reasonable number, has a class average <80% and

where the minimum number of EEIC is at least 8,

this parameter has been nearly 100% accurate in

finding highly suspicious pairs who have proximate

seating. Further we recommend discarding questions

where the most chosen answer exceeds the 'correct'

answer.

 

I am aware of the fact the Wesolowsky has often carried "HH" statistics along in the output created by SCheck, something I assume he may do as a sort of crosscheck.

 

We exchanged a fair number of email messages as part of my work.  Here is an extract from one of them:

 

The HH method is good as long as it stays on its reservation. For example,  it is ok if the number of questions is around 50 but HH generally underestimates similarity when there are 100  questions or so. It tends to overestimate when there is a small number of questions. There are also problems with calculating significance probabilities, as I wrote to you long ago.

 

The problem with HH significance testing relates to the claim by Harpp and Hogan that their test statistic, "sigma", has a log(Normal) distribution.  It almost certainly does not.

 

However, I have found that there is usually a great deal of overlap between the findings produced by SCheck and Lertap 5's RSA, and I continue to suggest that Lertap's RSA is an adequate tool for looking for possible cheating, keeping in mind, above all, the comments by Professor Harpp repeated above.

 

I also continue to compare the output from SCheck with that from Lertap RSA whenever possible. To make such comparisons easier, Lertap RSA now creates a copy of its RSAcases report called "RSAcasesNosort".

 

The entries in RSAcasesNosort are presented in very much the same order as are the cases in SCheck's output. That is, they're not sorted on the basis of HH sigma.

 

The formatting of each RSAcasesNosort report is fairly primitive. Because of this, I have Excel hide these reports from view -- but they're there for all to see: just use the "Unhide" option in Excel. And write to me for support, if desired.

 

Professor Wesolowsky, like me, is now semi-retired.  But (again like myself) he maintains an active interest in professional activities and is very approachable. He is familiar with Lertap RSA and knows that it produces a data file suitable for immediate use by SCheck.

 

Here is his main contact page as of April 2016:

 

http://www.business.mcmaster.ca/msis/profs/wesolo/wesolo.htm