Purpose: To assess external validity of a framework, which was previously developed to predict the incidence of radiation‐induced pneumontis in patients following radiotherapy for lung cancer, by testing it on an alternate dataset (other than the one used to develop it). Methods: The study employed two independent datasets: 1) a Duke University dataset of 219 patients, of whom 34 were diagnosed with post‐radiotherapy pneumonitis; and 2) a Washington University (WashU) dataset of 219 patients, of whom 52 developed pneumonitis post‐radiotherapy. While the Duke dataset was used to develop the framework, the WashU dataset was used to assess its validity. The framework was based on a predictive model that combined results from four machine learning models into a single fused model using a Bayesian decision fusion methodology. The results of testing the framework on the WashU dataset was measured in terms of the area under ROC curve (AUC) and compared to the AUC value obtained by Mean Lung Dose (MLD) — a metric commonly used to discriminate between patients with and without pneumonitis. Results: The framework yielded an overall AUC of 0.84 for cross‐validated testing on Duke data. Six patient features were identified as most significant. They consisted of two dose features: EUD (a=1.3) and V30; and four non‐dose features: female gender, tumor in central lung location, adenocarcinoma, and chemotherapy prior to radiotherapy. Most notably, when tested on the WashU dataset, the framework yielded an AUC value of 0.73. In comparison, the AUC value obtained by employing MLD values was only 0.68. Conclusions: Based on testing with an independent dataset, the framework to predict the incidence of radiation‐induced pneumonitis appears to be fairly robust. The framework may potentially be used in the clinic to improve the treatment decision‐making process.