Background Primary care physicians play a leading role in counselling older drivers, but discussions often do not occur until safety concerns arise. Prior work suggests that routine questioning about driving might facilitate these difficult conversations. Objective To explore system-level factors affecting driving discussions in primary care settings, in order to inform the design and implementation of a programme supporting routine conversations. Methods This qualitative descriptive study used iterative interviews with providers (physicians, nurses, medical assistants, social workers, and administrative staff) working at two clinics (one geriatric, one general internal medicine) at a tertiary-care teaching hospital. General inductive techniques in transcript analysis were used to identify stakeholder-perceived system-level barriers and facilitators to routine conversations with older drivers. Results From 15 interviews, four themes emerged: (1) complexity of defined provider roles within primary care setting (which can both support team work and hamper efficiency); (2) inadequate resources to support providers (including clinical prompts, local guides, and access to social workers and driving specialists); (3) gaps in education of providers and patients about discussing driving; and (4) suggested models to enhance provider conversations with older drivers (including following successful examples and using defined pathways integrated into the electronic medical record). A fifth theme was that participants characterised their experiences in terms of current and ideal states. Conclusions Physicians have been tasked with assessing older driver safety and guiding older patients through the process of ‘driving retirement.’ Attention to system-level factors such as provider roles, resources, and training can support them in this process.