Poor inter-rater reliability on mock anesthesia oral examinations

Purpose: Inter-rater reliability (IRR) and communication skills are both important factors that have been shown to affect oral examination scores. This study was designed to test: 1) IRR of a group of American Board of Anesthesiology (ABA) and the Royal College of Physicians and Surgeons of Canada (RCPSC) graders who graded in true isolation; 2) the effect of teaching residents examination techniques. Methods: This was a randomized, pretest-posttest trial. Twenty-five residents did an initial oral examination (E1) resembling the ABA examination. They were then randomized into two groups, a routine education group, and an intervention group that was taught oral examination skills. Six weeks later they did another oral examination (E2). The videotaped examinations were subsequently scored by six experienced RCPSC and ABA graders. Results: There was very poor IRR on E1 (weighted Kappa = 0.166, intraclass correlation coefficient 0.243), which improved only slightly on E2 (weighted Kappa = 0.275, P = NS; intraclass correlation coefficient = 0.405, P < 0.01). Pass rate for grader-pairs increased from E1 to E2 (15% vs 43%, P = 0.01). The improved pass rate on E2 occurred in both the routine education group and in the intervention group. There was no significant difference between RCPSC and ABA graders. Teaching examination skills per se did not improve performance, but this conclusion may be limited by the poor IRR. Practice orals do appear to improve performance on future examinations. Conclusions: Inter-rater reliability may be poor when graders score an oral examination in true isolation. Teaching candidates an oral examination communication and presentation technique did not appear to improve performance. Oral examination practice may be of value in training for future examinations.