Advanced Measurement Theory (5 weeks with Larry)

 

Overview of Major Objectives  for 2016 / 2559

 

The "measurement" we will discuss involves the use of tests and scales, often referred to as "instruments". We use them to collect information from people.

 

Exams, tests, and quizzes are referred to as cognitive instruments as they have to do with assessing cognition, where cognition may be defined as the psychological result of perception, learning, and reasoning. These instruments ask questions which are scored -- people get points when they answer a question correctly.

 

Scales, on the other hand, have to do with the "affective domain". Two examples would be "Job Satisfaction Inventory" and "Happiness Scale". Questions used on scales do not have correct answers.

 

My own work is focused on the development and use of cognitive instruments, and this work will form the subject of most of my presentations.  However, many of the methods and statistics you will see me using also apply to affective instruments, and we will have time to examine at least one popular affective scale.

 

(1) Classical Test Theory, "CTT"

 

We will use Lertap 5 for learning about CTT.

 

Lertap 5 is an Excel “app”, a sophisticated data processing system used for item and test analysis, and for test scoring. It is based on CTT, classical test theory, and is often used before undertaking an IRT analysis. We will use Lertap 5 to review CTT, look at classification consistency, investigate differential item functioning (DIF), look for test and item score differences among groups, see the usefulness of principal components analysis, and, as time permits, see how to get Lertap 5 to set up data files for use by IRT programs (EIRT, SAS, Xcalibre, and Bilog-MG).

 

(2) Item Response Theory, "IRT"

 

A special "Add-In" for Excel called EIRT will be one of the main resources for looking at how to use IRT.  If there is time, we may also look at the IRT capabilities found in the free version of SAS.

 

Some of the most "professional" IRT programs (expensive) are Xcalibre, from Assessment Systems Corporation, and programs from Scientific Software International, such as Bilog-MG, and IRTPRO.  I will give examples of using these programs as time permits.

 

Our discussion of IRT will take us into "CAT", computerized adaptive testing.  Two staff members at RMCS have recently completed CAT projects, and perhaps we can get one of them to discuss their research in class.

 

(3) Data processing experience

 

We will use an Excel-based CTT (Classical Test Theory) program called “Lertap 5”, an IRT (Item Response Theory) program called “EIRT”, and, if time permits, look at the IRT routines in SAS, Xcalibre, and Bilog-MG.

 

(4) Mention possible research ideas and projects

 

For suggestions, please refer to this topic.