Interrater reliability of surveillance for ventilator-associated events and pneumonia

OBJECTIVE To compare interrater reliabilities for ventilator-associated event (VAE) surveillance, traditional ventilator-associated pneumonia (VAP) surveillance, and clinical diagnosis of VAP by intensivists. DESIGN A retrospective study nested within a prospective multicenter quality improvement study. SETTING Intensive care units (ICUs) within 5 hospitals of the Centers for Disease Control and Prevention Epicenters. PATIENTS Patients who underwent mechanical ventilation. METHODS We selected 150 charts for review, including all VAEs and traditionally defined VAPs identified during the primary study and randomly selected charts of patients without VAEs or VAPs. Each chart was independently reviewed by 2 research assistants (RAs) for VAEs, 2 hospital infection preventionists (IPs) for traditionally defined VAP, and 2 intensivists for any episodes of pulmonary deterioration. We calculated interrater agreement using κ estimates. RESULTS The 150 selected episodes spanned 2,500 ventilator days. In total, 93-96 VAEs were identified by RAs; 31-49 VAPs were identified by IPs, and 29-35 VAPs were diagnosed by intensivists. Interrater reliability between RAs for VAEs was high (κ, 0.71; 95% CI, 0.59-0.81). Agreement between IPs using traditional VAP criteria was slight (κ, 0.12; 95% CI, -0.05-0.29). Agreement between intensivists was slight regarding episodes of pulmonary deterioration (κ 0.22; 95% CI, 0.05-0.39) and was fair regarding whether episodes of deterioration were attributable to clinically defined VAP (κ, 0.34; 95% CI, 0.17-0.51). The clinical correlation between VAE surveillance and intensivists' clinical assessments was poor. CONCLUSIONS Prospective surveillance using VAE criteria is more reliable than traditional VAP surveillance and clinical VAP diagnosis; the correlation between VAEs and clinically recognized pulmonary deterioration is poor.