NSQIP Quality Benchmarking and Evaluation of Potential Bias Associated with Higher- or Lower-Risk Operation Case Mix

Background: To ensure validity and acceptance of NSQIP risk-adjusted benchmarking, it is important that adjustments adequately control for hospitals that vary in their proportions of lower- or higher-risk operations (combined risk for procedure and patient). This issue was addressed in separate empirical and simulation studies. Study Design: For the empirical study, potential miscalibration bias favoring hospitals that do lower-risk operations or disfavoring hospitals that do higher-risk operations was evaluated for 14 modeled outcomes using NSQIP data. A determination was also made as to whether there was a relationship between mean hospital operation risk and benchmarking results (log odds ratio). In the simulation study of the same 14 outcomes, hospital benchmarked performance was evaluated when sampled cases were reconstituted to include either a larger proportion of lower-risk operations or a larger proportion of higher-risk operations. Results: Miscalibration favoring either lower- or higher-risk operations was absent, as were important associations between operative risk and hospital log odds ratios (most model R2 less than 0.01). In the simulation, there were no substantial changes in log odds ratios when greater percentages of either lower- or higher-risk operations were included in a hospital's sample (nonsignificant p values and effect sizes less than 0.1). Conclusions: These results should enhance NSQIP participants' confidence in the adequacy of NSQIP patient and procedure risk-adjustment methods. NSQIP participants may rely on benchmarking findings, and implement quality improvement efforts based on them, without concern that they are biased by a preponderance of lower or higher risk operations.