Background: High-level physical activity is associated with lower colorectal cancer (CRC) mortality, likely through insulin sensitization. Insulin receptor substrate 1 (IRS1) is a mediator of insulin and insulin-like growth factor (IGF) signaling pathways, and its down-regulation is associated with insulin resistance. Therefore, we hypothesized that tumor IRS1 expression status might modify cellular sensitivity to insulin and IGF, and the prognostic association of physical activity. Methods: We assessed IRS1 expression level in 371 stage I–III rectal and colon cancers in the Nurses’ Health Study and the Health Professionals Follow-up Study by immunohistochemistry. In survival analysis, Cox proportional hazards model was used to assess an interaction between post-diagnosis physical activity (ordinal scale of sex-specific quartiles Q1 to Q4) and IRS1 expression (ordinal scale of negative, low, and high), controlling for potential confounders, including microsatellite instability, CpG island methylator phenotype, long interspersed nucleotide element-1 (LINE-1) methylation level, and KRAS, BRAF, and PIK3CA mutation status. Results: There was a statistically significant interaction between post-diagnosis physical activity and tumor IRS1 expression in CRC-specific mortality analysis (Pinteraction = 0.005). Multivariable hazard ratio (95 % confidence interval) for higher post-diagnosis physical activity (Q3–Q4 vs. Q1–Q2) was 0.15 (0.02–1.38) in the IRS1-negative group, 0.45 (0.19–1.03) in the IRS1-low group, and 1.32 (0.50–3.53) in the IRS1-high group. Conclusions: The association of post-diagnosis physical activity with colorectal carcinoma patient survival may differ by tumor IRS1 expression level. If validated, tumor IRS1 expression status may serve as a predictive marker to identify subgroups of patients who might gain greater survival benefit from an increased level of exercise.