A long-held view is that stroke causes many distinct neurological syndromes due to damage of specialized cortical and subcortical centers. However, it is unknown if a syndrome-based description is helpful in characterizing behavioral deficits across a large number of patients. We studied a large prospective sample of first-time stroke patients with heterogeneous lesions at 1-2weeks post-stroke. We measured behavior over multiple domains and lesion anatomy with structural MRI and a probabilistic atlasof white matter pathways. Multivariate methods estimated the percentage of behavioral variance explained by structural damage. A few clusters of behavioral deficits spanning multiple functions explained neurological impairment. Stroke topography was predominantly subcortical, and disconnection of white matter tracts critically contributed to behavioral deficits and their correlation. The locus of damage explained more variance for motor and language than memory or attention deficits. Our findings highlight the need for better models of white matter damage on cognition.