Evaluating the Quality of AI-Written Scenarios for Virtual Oral Surgical Board Preparatory Examination

Objective: The objective of this study was to evaluate whether ChatGPT can generate level-appropriate clinical scenarios that are suitable for use in oral board preparatory examination (mock oral exam) for senior surgical residents. Design: This was a prospective, blinded study in which AI-written and faculty-written scenarios were reviewed, randomized and used for testing in virtual mock oral exam. Both faculty examiners and test-taking residents were blinded to the true authorship of the scenarios. After the examination, participants completed a survey evaluating the complexity of each scenario and their perceptions of its authorship. Setting: The study was conducted at Washington University in St. Louis (WashU), an academic medical center located in St. Louis, Missouri. The participating institutions also included Saint Louis University (SLU). Participants: Study participants included twenty-five senior general surgery residents (PGY4 and PGY5) and twenty faculty examiners from WashU and SLU, who took part in virtual mock oral examination. Post-exam surveys were completed both residents and faculty. Results: Faculty rated most AI-written and faculty written scenarios as "level-appropriate” in terms of both the quality of the text and the degree of complexity. Similarly, when residents were asked to identify the most difficult scenarios, they selected both AI- and faculty-written scenarios at comparable rates. Notably, both faculty and residents struggled to correctly distinguish the origin of the scenarios, with frequent misidentification across both groups. Conclusion: AI-written clinical scenarios were comparable to faculty-written scenarios in terms of complexity and appropriateness for senior surgical residents when used in a virtual mock oral board examination, highlighting the potential utility of AI-based tools in oral board preparation and surgical education.