Wolfram syndrome is a rare disease caused by mutations in the WFS1 gene leading to symptoms in early to mid-childhood. Brain structural abnormalities are present even in young children, but it is not known when these abnormalities arise. Such information is critical in determining optimal outcome measures for clinical trials and in understanding the aberrant neurobiological processes in Wolfram syndrome. Using voxel-wise and regional longitudinal analyses, we compared brain volumes in Wolfram patients (n = 29; ages 5–25 at baseline; mean follow-up = 3.6 years), to age and sex-equivalent controls (n = 52; ages 6–26 at baseline; mean follow-up = 2.0 years). Between groups, white and gray matter volumes were affected differentially during development. Controls had uniformly increasing volume in white matter, whereas the Wolfram group had stable (optic radiations) or decreasing (brainstem, ventral pons) white matter volumes. In gray matter, controls had stable (thalamus, cerebellar cortex) or decreasing volumes (cortex), whereas the Wolfram group had decreased volume in thalamus and cerebellar cortex. These patterns suggest that there may be early, stalled white matter development in Wolfram syndrome, with additional degenerative processes in both white and gray matter. Ideally, animal models could be used to identify the underlying mechanisms and develop specific interventions.