Measuring content overlap during handoff communication using distributional semantics: An exploratory study

Objective We develop and evaluate a methodological approach to measure the degree and nature of overlap in handoff communication content within and across clinical professions. This extensible, exploratory approach relies on combining techniques from conversational analysis and distributional semantics. Materials and methods We audio-recorded handoff communication of residents and nurses on the General Medicine floor of a large academic hospital (n = 120 resident and n = 120 nurse handoffs). We measured semantic similarity, a proxy for content overlap, between resident-resident and nurse-nurse communication using multiple steps: a qualitative conversational content analysis; an automated semantic similarity analysis using Reflective Random Indexing (RRI); and comparing semantic similarity generated by RRI analysis with human ratings of semantic similarity. Results There was significant association between the semantic similarity as computed by the RRI method and human rating (ρ = 0.88). Based on the semantic similarity scores, content overlap was relatively higher for content related to patient active problems, assessment of active problems, patient-identifying information, past medical history, and medications/treatments. In contrast, content overlap was limited on content related to allergies, family-related information, code status, and anticipatory guidance. Conclusions Our approach using RRI analysis provides new opportunities for characterizing the nature and degree of overlap in handoff communication. Although exploratory, this method provides a basis for identifying content that can be used for determining shared understanding across clinical professions. Additionally, this approach can inform the development of flexibly standardized handoff tools that reflect clinical content that are most appropriate for fostering shared understanding during transitions of care.