Machine learning pipelines developed for the prediction of cancelation of inappropriate parathyroid hormone-related peptide orders demonstrate poor performance in predicting provider behavior

Background: Quantification of circulating parathyroid hormone-related peptide (PTHrP) aids in the diagnosis of humoral hypercalcemia of malignancy. However, utilization of this test in the setting of low pretest probability leads to false positive results, unnecessary follow-up testing, and patient anxiety. As part of an initiative to improve laboratory utilization, all PTHrP orders at our institution are reviewed by a laboratory medicine resident (LMR), who contacts the ordering physician when an order is deemed to have low utility. This review process is time- and labor-intensive, and may sow discontent with providers who feel they are being “second guessed”. We sought to apply machine learning to automate this review process and minimize futile LMR interventions. Methods: Retrospective, first-time PTHrP orders from 2019 to 2022 (n=1,144) were extracted from the laboratory information system of a single healthcare system. The dataset was partitioned into an 80:20 split between training and testing sets. XGBoost models were trained to predict order cancelation and order result, using laboratory data available at the time of the PTHrP order as features. After training and cross-validation, the models were applied to the held out test set and performance was evaluated using area under the receiver operating characteristic curve (AUC_ROC). Results: Six hundred and forty-two (56%) PTHrP orders were ordered on patients without a recently suppressed PTH (<25 pg/mL), while 467 (41%) were placed on patients without recent hypercalcemia (>11 mg/dL). Of these, 450 were not canceled and only 9 (2%) were positive. The model trained to predict whether a PTHrP order was completed or canceled demonstrated little discriminatory power, with an AUC_ROC of 0.64 [95% confidence interval (CI): 0.59–0.58]. However, a model trained using the same pipeline to instead predict the PTHrP result demonstrated an AUC_ROC of 0.85 (95% CI: 0.82–0.87). Conclusions: The performance difference between the models trained on the two different targets suggests that the physician’s willingness to cancel in response to the LMR-driven intervention may be unrelated to the diagnostic value of the test in the context of other laboratory data.