Objective: To investigate whether longitudinal declines in cognition are associated with higher fibrillar amyloid-beta (Aβ) deposition in vivo in individuals without dementia. Method: [C]PiB images were obtained to measure fibrillar Aβ burden in 57 participants without dementia from the Baltimore Longitudinal Study of Aging. Participants (33 men, 24 women) had a mean (SD) age of 78.7 (6.2) years. Six participants (4 men, 2 women) had mild cognitive impairment defined as Clinical Dementia Rating = 0.5. To measure [C]PiB retention, distribution volume ratios (DVR) for 15 regions of interest were estimated by fitting a simplified reference tissue model to the measured time activity curves. Mixed effects regression was used to predict cognitive trajectories over time using data before and including time of PiB (mean follow-up 10.8 years), with mean cortical DVR, age at baseline, sex, and education as independent predictors. Voxel-based analysis identified local associations. Results: [C]PiB retention was higher in older individuals. Greater declines over time in mental status and verbal learning and memory, but not visual memory, were associated significantly with higher PiB retention. Voxel-based analysis showed significant associations in frontal and lateral temporal regions. Conclusions: Higher Aβ deposition is associated with greater longitudinal decline in mental status and verbal memory in the preceding years. The differential association for verbal but not visual memory may reflect the greater reliance of verbal word list learning on prefrontal regions, which show early Aβ deposition. Prospective imaging may help distinguish between individuals with evolving neuropathology who develop accelerated cognitive decline vs those with normal aging.