The Invariance of Item Response Theory (IRT) Parameter Estimates and Classical Test Theory (CTT) Statistics

The paper intends to investigate the invariance of the parameter estimates generated from Item Response Theory (IRT) models and those generated from the Classical Test Theory (CTT) model. There have been problems associated the Classical Test Theory or number-right scoring system used by many assessment bodies in Africa. These problems include the need for parallel test forms each year which themselves are not easy to construct. The statistics from CTT are examinee dependent and the performance on a test depends on the ability of examinees. This study investigated the invariance of CTT item/person statistics and IRT item/person parameter estimates. It has been assumed rather casually that IRT parameter estimates are more reliable to CTT item/person statistics. Recent research (for example, Fan, 1998) paint a different picture that the two assessment framework actually yield estimates that are comparable than previously thought. The item and person statistics and parameter estimates for the tests are compared to establish whether there exists a significant difference amongst them. The sample of the study consist of about 2000 Form Three (Grade 10) examinees that were about to sit for their final JCE examinations in 2008. The other data set was the national data from the 2005 and 2006 cohort. The instruments were the 2005 and the 2006 Botswana Junior Certificate objective science papers.

The study intended to test the Invariance of Item/Person Parameter Estimates for CTT and IRT. The CTT p-values (difficulty) and their corresponding IRT b-parameter estimates are highly correlated for all the IRT models. For the CTT item discrimination index a, (item-test, point-biserial correlation-rpbs) and the IRT item discrimination a-parameter (item slope parameter), the two statistics are highly correlated for the 1-PL and 2-PLbut not for the IRT 3-PL model. Notwithstanding this, the CTT item/person statistics and IRT item/person parameter estimates are invariant and similar conclusions could be drawn irrespective of which method was used to estimate the examinee's ability.