Background: Intratumor heterogeneity implies that subpopulations of cancer cells that differ in genetic, phenotypic, or behavioral characteristics coexist in a single tumor (Ma in Breast Cancer Res Treat 162(1):39–48, 2017; Martelotto in Breast Cancer Res 16(3):210, 2014). Tumor heterogeneity drives progression, metastasis and treatment resistance, but its relationship with tumor infiltrating immune cells is a matter of debate, where some argue that tumors with high heterogeneity may generate neoantigens that attract immune cells, and others claim that immune cells provide selection pressure that shapes tumor heterogeneity (McGranahan et al. in Science 351(6280):1463–1469, 2016; McGranahan and Swanton in Cell 168(4):613–628, 2017). We sought to study the association between tumor heterogeneity and immune cells in a real-world cohort utilizing The Cancer Genome Atlas. Methods: Mutant allele tumor heterogeneity (MATH) was calculated to estimate intratumoral heterogeneity, and immune cell compositions were estimated using CIBERSORT. Survival analyses were demonstrated using Kaplan–Meir curves. Results: Tumors with high heterogeneity (high MATH) were associated with worse overall survival (p = 0.049), as well as estrogen receptor-positive (p = 0.011) and non-triple-negative tumors (p = 0.01). High MATH tumors were also associated with less infiltration of anti-tumor CD8 (p < 0.013) and CD4 T cells (p < 0.00024), more tumor-promoting regulatory T cells (p < 4e−04), lower expression of T-cell exhaustion markers, specifically PDL-1 (p = 0.0031), IDO2 (p = 0.34), ADORA2A (p = 0.018), VISTA (p = 0.00013), and CCR4 (p < 0.00001), lower expression of cytolytic enzymes granzyme A (p = 0.0056) and perforin 1 (p = 0.053), and low cytolytic activity score (p = 0.0028). Conclusions: High heterogeneity tumors are associated with less immune cell infiltration, less activation of the immune response, and worse survival in breast cancer. Our results support the notion that tumor heterogeneity is shaped by selection pressure of tumor-infiltrating immune cells.