An Examination of American College of Surgeons NSQIP Surgical Risk Calculator Accuracy

Mark E. Cohen, Yaoming Liu, Clifford Y. Ko, Bruce L. Hall

Research output: Contribution to journalArticlepeer-review

83 Scopus citations

Abstract

Background The American College of Surgeons NSQIP offers a Surgical Risk Calculator (SRC) that provides detailed, patient-level, risk assessments for many adverse outcomes to surgeons, patients, and the general public. The SRC calculator was designed to help guide discussion and decisions by providing generally applicable (not hospital-specific) information about surgical risk using easily understood and broadly available preoperative variables. Although large, internal evaluations have shown that the SRC has good accuracy (model discrimination and calibration), external validations have been inconsistent and tend to favor a conclusion of inadequate performance. Study Design External studies, attempting to validate the SRC, were examined with respect to 3 design features: sample size (small samples reduce reliability), case-mix homogeneity (homogeneity reduces discrimination); and number of institutions providing data (few institutions reduces generalizability). The impact of each feature was then examined in several sets of simulation studies. Results Each of the 3 design features has the potential to act as an artifactual cause for apparent SRC predictive failure. In addition, demonstrations that SRC estimates are inferior to those from models that use additional (sometimes operation-specific) predictor variables were seen as not relevant with respect to the SRC's intended scope. Conclusions The SRC predictive failures, reported by studies with the described design limitations, should not be misunderstood as disqualifying the SRC as an accurate and appropriate tool for its intended purpose of providing a general purpose risk calculator, applicable across many surgical domains, using easily understood and generally available predictive information.

Original languageEnglish
Pages (from-to)787-795.e1
JournalJournal of the American College of Surgeons
Volume224
Issue number5
DOIs
StatePublished - May 2017

Fingerprint

Dive into the research topics of 'An Examination of American College of Surgeons NSQIP Surgical Risk Calculator Accuracy'. Together they form a unique fingerprint.

Cite this