To assess the directionality of the association between physical and cognitive decline in later life, we compared patterns of decline in performance across groups defined by baseline presence of cognitive and/or physical impairment [none (n = 217); physical only (n = 169); cognitive only (n = 158), or both (n = 220)] in a large sample of participants in a cognitive aging study at the Knight Alzheimer's Disease Research Center at Washington University in St. Louis who were followed for up to 8 years (3,079 observations). Rates of decline reached 20% for physical performance and varied across cognitive tests (global, memory, speed, executive function, and visuospatial skills). We found that physical decline was better predicted by baseline cognitive impairment (slope = -1.22, p<0.001), with baseline physical impairment not contributing to further decline in physical performance (slope = -0.25, p = 0.294). In turn, baseline physical impairment was only marginally associated with rate of cognitive decline across various cognitive domains. The cognitive-functional association is likely to operate in the direction of cognitive impairment to physical decline although physical impairment may also play a role in cognitive decline/dementia. Interventions to prevent further functional decline and development of disability and complete dependence may benefit if targeted to individuals with cognitive impairment who are at increased risk.