Children with neurofibromatosis type 1 (NF1) develop low-grade brain tumors throughout the optic pathway. Nearly 50% of children with optic pathway gliomas (OPGs) experience visual impairment, and few regain their vision after chemotherapy. Recent studies have revealed that girls with optic nerve gliomas are five times more likely to lose vision and require treatment than boys. To determine the mechanism underlying this sexually dimorphic difference in clinical outcome, we leveraged Nf1 optic glioma (Nf1-OPG) mice. We demonstrate that female Nf1-OPG mice exhibit greater retinal ganglion cell (RGC) loss and only females have retinal nerve fiber layer (RNFL) thinning, despite mice of both sexes harboring tumors of identical volumes and proliferation. Female gonadal sex hormones are responsible for this sexual dimorphism, as ovariectomy, but not castration, of Nf1-OPG mice normalizes RGC survival and RNFL thickness. In addition, female Nf1-OPG mice have threefold more microglia than their male counterparts, and minocycline inhibition of microglia corrects the retinal pathology. Moreover, pharmacologic inhibition of microglial estrogen receptor-β (ERβ) function corrects the retinal abnormalities in female Nf1-OPG mice. Collectively, these studies establish that female gonadal sex hormones underlie the sexual dimorphic differences in Nf1 optic glioma-induced retinal dysfunction by operating at the level of tumor-associated microglial activation.