Patients with sickle cell anemia (SCA) experience cerebral metabolic stress with an increase in oxygen extraction fraction (OEF) to compensate for reduced oxygen carrying capacity due to anemia. It remains unclear if anemia alone drives this metabolic stress. Using MRI, we collected voxel-wise OEF measurements to test our hypothesis that OEF would be elevated in anemic controls without SCA (AC) compared to healthy controls (HC), but OEF would be even higher in SCA compared to AC. Brain MRIs (N = 159) were obtained in 120 participants (34 HC, 27 AC, 59 SCA). While hemoglobin was lower in AC versus HC (p < 0.001), hemoglobin was not different between AC and SCA cohorts (p = 0.459). Whole brain OEF was higher in AC compared to HC (p < 0.001), but lower compared to SCA (p = 0.001). Whole brain OEF remained significantly higher in SCA compared to HC (p = 0.001) while there was no longer a difference between AC versus HC (p = 0.935) in a multivariate model controlling for age and hemoglobin. OEF peaked within the border zone regions of the brain in both SCA and AC cohorts, but the volume of white matter with regionally elevated OEF in AC was smaller (1.8%) than SCA (58.0%). While infarcts colocalized within regions of elevated OEF, more SCA participants had infarcts than AC (p < 0.001). We conclude that children with SCA experience elevated OEF compared to AC and HC after controlling for the impact of anemia, suggesting that there are other pathophysiologic factors besides anemia contributing to cerebral metabolic stress in children with SCA.