Evaluating the quality, clinical relevance, and resident perception of the radiation oncology in-training examination: A national survey

Purpose: The yearly radiation oncology in-training examination (ITE) by the American College of Radiology is a widely used, norm-referenced educational assessment, with high test reliability and psychometric performance. We distributed a national survey to evaluate the academic radiation oncology community's perception of the ITE. Methods and Materials: In June 2014, a 7-question online survey was distributed via e-mail to current radiation oncology residents, program directors, and attending physicians who had completed residency in the past 5 years or junior attendings. Survey questions were designed on a 5-point Likert scale. Sign test was performed with P ≤ .05 considered statistically different from neutral. Results: Thirty-one program directors (33.3%), 114 junior attendings (35.4%), and 225 residents (41.2%) responded. Junior attendings and program directors reported that the ITE directly contributed to their preparation for the American Board of Radiology written certification (. P = .050 and .004, respectively). Residents did not perceive the examination as an accurate assessment of relevant clinical and scientific knowledge (. P < .0001) and feel the quality assurance is insufficient in its current form (. P < .0001). Residents and junior attendings agree that there are factual errors, and unclear questions/answers (. P < .0001 and .04, respectively). Free response suggestions included: less questions on rare disease sites (16.4%), more relevance to clinical practice (15.4%), avoiding questions that discriminate between a few percentage points (11.8%), and designing the test similar to the written certification examination (9.2%). Conclusions: Despite high examination reliability and psychometric performance, resident and attending physicians report a need for improved quality assurance and clinical relevance in the ITE. Although the current examination allows limited feedback, establishing a venue for individualized feedback may allow continual and timely improvement of the ITE. Adopting a criterion-referenced examination may further increase resident investment in and utilization of this valuable learning tool.