Reliability of an Essay Question – Effect of Using an Answer Key

Essay questions (EQ) and short essay questions (SEQ) are commonly used to evaluate higher order of the cognitive domain in medical education. However their reliability is poor due to high inter-rater and intra-rater variability. The aim of this study is to evaluate the reliability of EQ and SEQ and the role of using an answer key in improving reliability.

In this prospective comparative study, five examiners with different levels of experience (Examiner 1, 2 years of experience, Examiners 2 and 3: 6–7 years, Examiner 4: 15 years and Examiner 5: 18 years) evaluated 59 answer scripts with and without using the answer key. The answer key was prepared by a sixth participant (with 18 years’ experience) in the study using a standard textbook. Intra-rater variability and inter-rater variability were evaluated using paired ‘t’ test and ANOVA. Multivariate analysis was done using two-way ANOVA.

There was statistically significant intra-rater variability when evaluation was done with and without the key for all Examiners (p = 0.006, p < 0.001, p = 0.001, p = 0.001 for examiners 1–4, respectively) except for Examiner 5 (p = 0.19). There was statistically significant inter-rater variability between the examiners both for EQ (p = 0.004) and SEQ (p = 0.05), which reduced on using the key (EQ, p = 0.01, SEQ, p = 0.25). Use of key independently reduced the inter-rater variability on multivariate analysis both for EQ (p = 0.001) and SEQ (p = 0.007)

This study has shown that EQ and SAQ are not very reliable tools of evaluation. The reliability improved with the use of answer key and with the increase of experience of examiners.