TY - JOUR
T1 - Effect of machine learning models on clinician prediction of postoperative complications
T2 - the Perioperative ORACLE randomised clinical trial
AU - Fritz, Bradley A.
AU - King, Christopher R.
AU - Abdelhack, Mohamed
AU - Chen, Yixin
AU - Kronzer, Alex
AU - Abraham, Joanna
AU - Tripathi, Sandhya
AU - Ben Abdallah, Arbi
AU - Kannampallil, Thomas
AU - Budelier, Thaddeus P.
AU - Helsten, Daniel
AU - Montes de Oca, Arianna
AU - Mehta, Divya
AU - Sontha, Pratyush
AU - Higo, Omokhaye
AU - Kerby, Paul
AU - Gregory, Stephen H.
AU - Wildes, Troy S.
AU - Avidan, Michael S.
N1 - Publisher Copyright:
© 2024 British Journal of Anaesthesia
PY - 2024/11
Y1 - 2024/11
N2 - Background: Anaesthesiologists might be able to mitigate risk if they know which patients are at greatest risk for postoperative complications. This trial examined the impact of machine learning models on clinician risk assessment. Methods: This single-centre, prospective, randomised clinical trial enrolled surgical patients aged ≥18 yr. Anaesthesiologists and nurse anaesthetists providing remote telemedicine support reviewed electronic health records with (assisted group) or without (unassisted group) reviewing machine learning predictions. Clinicians predicted the likelihood of postoperative 30-day all-cause mortality and postoperative acute kidney injury (AKI) within 7 days. The primary outcome was area under the receiver operating characteristic curve (AUROC) for clinician predictions of mortality and AKI, comparing AUROCs between assisted and unassisted assessments. Results: We analysed 5071 patients (mean [range] age: 58 [18-100] yr; 52% female) assessed by 89 clinicians. Of these, 98 (2.2%) patients died within 30 days of surgery and 450 (11.1%) patients sustained AKI. Clinician predictions agreed with the models more strongly in the assisted vs unassisted group (weighted kappa 0.75 vs 0.62 for death, mean difference: 0.13 [95% CI 0.10–0.17]; and 0.79 vs 0.54 for AKI, mean difference: 0.25 [95% CI 0.21–0.29]). Clinical prediction of death was similar between the assisted (AUROC 0.793) and unassisted (AUROC 0.780) groups (mean difference: 0.013 [95% CI –0.070 to 0.097]; P=0.76). Prediction of AKI had an AUROC of 0.734 in the assisted group vs 0.688 in the unassisted group (difference 0.046 [95% CI –0.003 to 0.091]; P=0.06). Conclusions: Clinician performance was not improved by machine learning assistance. Further work is needed to clarify the role of machine learning in real-time perioperative risk stratification. Clinical trial registration: NCT05042804.
AB - Background: Anaesthesiologists might be able to mitigate risk if they know which patients are at greatest risk for postoperative complications. This trial examined the impact of machine learning models on clinician risk assessment. Methods: This single-centre, prospective, randomised clinical trial enrolled surgical patients aged ≥18 yr. Anaesthesiologists and nurse anaesthetists providing remote telemedicine support reviewed electronic health records with (assisted group) or without (unassisted group) reviewing machine learning predictions. Clinicians predicted the likelihood of postoperative 30-day all-cause mortality and postoperative acute kidney injury (AKI) within 7 days. The primary outcome was area under the receiver operating characteristic curve (AUROC) for clinician predictions of mortality and AKI, comparing AUROCs between assisted and unassisted assessments. Results: We analysed 5071 patients (mean [range] age: 58 [18-100] yr; 52% female) assessed by 89 clinicians. Of these, 98 (2.2%) patients died within 30 days of surgery and 450 (11.1%) patients sustained AKI. Clinician predictions agreed with the models more strongly in the assisted vs unassisted group (weighted kappa 0.75 vs 0.62 for death, mean difference: 0.13 [95% CI 0.10–0.17]; and 0.79 vs 0.54 for AKI, mean difference: 0.25 [95% CI 0.21–0.29]). Clinical prediction of death was similar between the assisted (AUROC 0.793) and unassisted (AUROC 0.780) groups (mean difference: 0.013 [95% CI –0.070 to 0.097]; P=0.76). Prediction of AKI had an AUROC of 0.734 in the assisted group vs 0.688 in the unassisted group (difference 0.046 [95% CI –0.003 to 0.091]; P=0.06). Conclusions: Clinician performance was not improved by machine learning assistance. Further work is needed to clarify the role of machine learning in real-time perioperative risk stratification. Clinical trial registration: NCT05042804.
KW - acute kidney injury
KW - anaesthesiology risk assessment
KW - artificial intelligence
KW - clinical trial
KW - machine learning
KW - postoperative complications
KW - postoperative death
UR - http://www.scopus.com/inward/record.url?scp=85203430672&partnerID=8YFLogxK
U2 - 10.1016/j.bja.2024.08.004
DO - 10.1016/j.bja.2024.08.004
M3 - Article
C2 - 39261226
AN - SCOPUS:85203430672
SN - 0007-0912
VL - 133
SP - 1042
EP - 1050
JO - British journal of anaesthesia
JF - British journal of anaesthesia
IS - 5
ER -