Leveraging Artificial Intelligence to Enhance Peer Review: Missed Liver Lesions on Computed Tomographic Pulmonary Angiography

Sarah P. Thomas, Tyler J. Fraum, Lawrence Ngo, Robert Harris, Elie Balesh, Mustafa R. Bashir, Benjamin Wildman-Tobriner

Research output: Contribution to journalArticlepeer-review

3 Scopus citations


Purpose: The aim of this study was to use artificial intelligence (AI) to facilitate peer review for detection of missed suspicious liver lesions (SLLs) on CT pulmonary angiographic (CTPA) examinations. Methods: This retrospective study included 1 month of consecutive CTPA examinations from a multisite teleradiology practice. Visual classification (VC) software analyzed images for the presence (+) or absence (−) of SLLs (>1 cm, >20 Hounsfield units). Separately, a natural language processing (NLP) algorithm evaluated corresponding reports for description (+) of an SLL or lack thereof (−). Studies containing possible missed SLLs (VC+/NLP−) were reviewed by three abdominal radiologists in a two-step adjudication process to confirm if an SLL was missed by the interpreting radiologist. The number of VC+/NLP− cases, the number of images needing radiologist review, and the number of cases with confirmed missed SLLs were recorded. Interobserver agreement for SLLs was calculated for the radiologist readers. Results: A total of 2,573 CTPA examinations were assessed, and 136 were classified as potentially containing missed SLLs (VC+/NLP−). After radiologist review, 13 cases with missed SLLs were confirmed, representing 0.5% of analyzed CT studies. Using AI, the ratio of CT studies requiring review to missed SLLs identified was 10:1; the ratio without the help of AI would be at least 66:1. Among the 136 cases reviewed by radiologists, interobserver agreement for SLLs was excellent (κ = 0.91). Conclusions: AI can accelerate meaningful peer review by rapidly assessing thousands of examinations to identify potentially clinically significant errors. Although radiologist involvement is necessary, the amount of effort required after initial AI screening is dramatically reduced.

Original languageEnglish
Pages (from-to)1286-1294
Number of pages9
JournalJournal of the American College of Radiology
Issue number11
StatePublished - Nov 2022


  • Peer review
  • artificial intelligence
  • liver lesion
  • radiology


Dive into the research topics of 'Leveraging Artificial Intelligence to Enhance Peer Review: Missed Liver Lesions on Computed Tomographic Pulmonary Angiography'. Together they form a unique fingerprint.

Cite this