While prior work reliably demonstrates that the APOE ϵ4 allele has deleterious group level effects on Alzheimer disease pathology, the homogeneity of its influence across the lifespan and spatially in the brain remains unknown. Further it is unclear what combinations of factors at an individual level lead to observed group level effects of APOE genotype. To evaluate the impact of the APOE genotype on disease trajectories, we examined longitudinal MRI and PET imaging in a cohort of 497 cognitively normal middle and older aged participants. A whole-brain regional approach was used to evaluate the spatial effects of genotype on longitudinal change of amyloid-β pathology and cortical atrophy. Carriers of the ϵ4 allele had increased longitudinal accumulation of amyloid-β pathology diffusely through the cortex, but the emergence of this effect across the lifespan differed greatly by region (e.g. age 49 in precuneus, but 65 in the visual cortex) with the detrimental influence already being evident in some regions in middle age. This increased group level effect on accumulation was due to a greater proportion of ϵ4 carriers developing amyloid-β pathology, on average doing so at an earlier age, and having faster amyloid-β accumulation even after accounting for baseline amyloid-β levels. APOE ϵ4 carriers displayed faster rates of structural loss in primarily constrained to the medial temporal lobe structures at around 50 years, although this increase was modest and proportional to the elevated disease severity in APOE ϵ4 carriers. This work indicates that influence of the APOE gene on pathology can be detected starting in middle age.