Objective White matter (WM) projections were assessed from Alzheimer disease (AD) gray matter regions associated with β-amyloid (Aβ), tau, or neurodegeneration to ascertain relationship between WM structural integrity with Aβ and/or tau deposition. Methods Participants underwent diffusion tensor imaging (DTI), PET Aβ ([18F]AV-45 [florbetapir]), and PET tau ([18F]AV-1451 [flortaucipir]) imaging. Probabilistic WM summary and individual tracts were created from either a composite or individual gray matter seed regions derived from Aβ, tau, and neurodegeneration. Linear regressions were performed for Aβ, age, tau and WM hyperintensities (WMH) to predict mean diffusivity (MD) or fractional anisotropy (FA) from the corresponding WM summaries or tracts. Results Our cohort was composed of 59 cognitively normal participants and 10 cognitively impaired individuals. Aβ was not associated with DTI metrics in WM summary or individual tracts. Age and WMH strongly predicted MD and FA in several WM regions, with tau a significant predictor of MD only in the anterior temporal WM. Conclusion Tau, not Aβ, was associated with changes in anterior temporal WM integrity. WMH, a proxy for vascular damage, was strongly associated with axonal damage, but tau independently contributed to the model, suggesting an additional degenerative mechanism within tracts projecting from regions vulnerable to AD pathology. WM decline was associated with early tau accumulation, and further decline may reflect tau propagation in more advanced stages of AD.