We present a label-free, chemically-selective, quantitative imaging strategy to identify breast cancer and differentiate its subtypes using coherent anti-Stokes Raman scattering (CARS) microscopy. Human normal breast tissue, benign proliferative, as well as in situ and invasive carcinomas, were imaged ex vivo. Simply by visualizing cellular and tissue features appearing on CARS images, cancerous lesions can be readily separated from normal tissue and benign proliferative lesion. To further distinguish cancer subtypes, quantitative disease-related features, describing the geometry and distribution of cancer cell nuclei, were extracted and applied to a computerized classification system. The results show that in situ carcinoma was successfully distinguished from invasive carcinoma, while invasive ductal carcinoma (IDC) and invasive lobular carcinoma were also distinguished from each other. Furthermore, 80% of intermediate-grade IDC and 85% of high-grade IDC were correctly distinguished from each other. The proposed quantitative CARS imaging method has the potential to enable rapid diagnosis of breast cancer.