Background Patients retained in HIV care but not on antiretroviral therapy (ART) represent an important part of the HIV care cascade in the United States. Even in an era of more tolerable and efficacious ART, decision making in regards to ART offer and uptake remains complex and calls for exploration of both patient and provider perspectives. We sought to understand reasons for lack of ART usage in patients meeting the Health Resources Services Administration definition of retention as well as what motivated HIV primary care appointment attendance in the absence of ART. Methods and Findings We conducted a qualitative study consisting of 70 in-depth interviews with ART-naïve and ART-experienced patients off ART and their primary care providers in two urban safety-net HIV clinics in San Francisco and New York. Twenty patients and their providers were interviewed separately at baseline, and 15 dyads were interviewed again after at least 3 mo and another clinic visit in order to understand any ART use in the interim. We applied dyadic analysis to our data. Nearly all patients were willing to consider ART, and 40% of the sample went on ART, citing education on newer antiretroviral drugs, acceptance of HIV diagnosis, social support, and increased confidence in their ability to adhere as facilitators. However, the strength of the provider recommendation of ART played an important role. Many patients had internalized messages from providers that their health was too good to warrant ART. In addition, providers, while demonstrating patient-centered care through sensitivity to patients experiencing psychosocial instability, frequently muted the offer of ART, at times unintentionally. In the absence of ART, lab monitoring, provider relationships, access to social services, opiate pain medications, and acute symptoms motivated care. The main limitations of this study were that treatment as prevention was not explored in depth and that participants were recruited from academic HIV clinics in the US, making the findings most generalizable to this setting. Conclusions Provider communication with regard to ART is a key focus for further exploration and intervention in order to increase ART uptake for those retained in HIV care.