Objective: Congenital heart disease (CHD) is associated with abnormal brain development in utero. We applied innovative fetal magnetic resonance imaging (MRI) techniques to determine whether reduced fetal cerebral substrate delivery impacts the brain globally, or in a region-specific pattern. Our novel design included two control groups, one with and the other without a family history of CHD, to explore the contribution of shared genes and/or fetal environment to brain development. Methods: From 2014 to 2018, we enrolled 179 pregnant women into 4 groups: “HLHS/TGA” fetuses with hypoplastic left heart syndrome (HLHS) or transposition of the great arteries (TGA), diagnoses with lowest fetal cerebral substrate delivery; “CHD-other,” with other CHD diagnoses; “CHD-related,” healthy with a CHD family history; and “optimal control,” healthy without a family history. Two MRIs were obtained between 18 and 40 weeks gestation. Random effect regression models assessed group differences in brain volumes and relationships to hemodynamic variables. Results: HLHS/TGA (n = 24), CHD-other (50), and CHD-related (34) groups each had generally smaller brain volumes than the optimal controls (71). Compared with CHD-related, the HLHS/TGA group had smaller subplate (−13.3% [standard error = 4.3%], p < 0.01) and intermediate (−13.7% [4.3%], p < 0.01) zones, with a similar trend in ventricular zone (−7.1% [1.9%], p = 0.07). These volumetric reductions were associated with lower cerebral substrate delivery. Interpretation: Fetuses with CHD, especially those with lowest cerebral substrate delivery, show a region-specific pattern of small brain volumes and impaired brain growth before 32 weeks gestation. The brains of fetuses with CHD were more similar to those of CHD-related than optimal controls, suggesting genetic or environmental factors also contribute. ANN NEUROL 2021;89:143–157.