Leveraging Artificial Intelligence to Enhance Peer Review: Missed Liver Lesions on Computed Tomographic Pulmonary Angiography

Purpose: The aim of this study was to use artificial intelligence (AI) to facilitate peer review for detection of missed suspicious liver lesions (SLLs) on CT pulmonary angiographic (CTPA) examinations. Methods: This retrospective study included 1 month of consecutive CTPA examinations from a multisite teleradiology practice. Visual classification (VC) software analyzed images for the presence (+) or absence (−) of SLLs (>1 cm, >20 Hounsfield units). Separately, a natural language processing (NLP) algorithm evaluated corresponding reports for description (+) of an SLL or lack thereof (−). Studies containing possible missed SLLs (VC+/NLP−) were reviewed by three abdominal radiologists in a two-step adjudication process to confirm if an SLL was missed by the interpreting radiologist. The number of VC+/NLP− cases, the number of images needing radiologist review, and the number of cases with confirmed missed SLLs were recorded. Interobserver agreement for SLLs was calculated for the radiologist readers. Results: A total of 2,573 CTPA examinations were assessed, and 136 were classified as potentially containing missed SLLs (VC+/NLP−). After radiologist review, 13 cases with missed SLLs were confirmed, representing 0.5% of analyzed CT studies. Using AI, the ratio of CT studies requiring review to missed SLLs identified was 10:1; the ratio without the help of AI would be at least 66:1. Among the 136 cases reviewed by radiologists, interobserver agreement for SLLs was excellent (κ = 0.91). Conclusions: AI can accelerate meaningful peer review by rapidly assessing thousands of examinations to identify potentially clinically significant errors. Although radiologist involvement is necessary, the amount of effort required after initial AI screening is dramatically reduced.