Neural progenitor cells undergo identity transitions during development to ensure the generation different types of neurons and glia in the correct sequence and proportions. A number of temporal identity factors that control these transitions in progenitor competence have been identified, but the molecular mechanisms underlying their function remain unclear. Here, we asked how Casz1, the mammalian orthologue of Drosophila castor, regulates competence during retinal development. We show that Casz1 is required to control the transition between neurogenesis and gliogenesis. Using BioID proteomics, we reveal that Casz1 interacts with the nucleosome remodeling and deacetylase (NuRD) complex in retinal cells. Finally, we show that both the NuRD and the polycomb repressor complexes are required for Casz1 to promote the rod fate and suppress gliogenesis. As additional temporal identity factors have been found to interact with the NuRD complex in other contexts, we propose that these factors might act through this common biochemical process to regulate neurogenesis.