Associations between brain structure and problematic alcohol use may reflect alcohol-induced toxicity and/or preexisting risk. Here, we applied a latent causal variable approach to genome-wide association study summary statistics of problematic alcohol use (n = 435,563) and magnetic resonance imaging-derived brain structure phenotypes (e.g., cortical volume, cortical thickness, white matter volume; ns ranging from 17,706 to 51,665) to test whether variability in brain structure may plausibly contribute to problematic alcohol use and/or whether problematic alcohol use influences brain structure. After correction for multiple testing within each modality, we find evidence that greater volume of the pars opercularis, greater thickness of the cuneus, and lower volume of the basal forebrain may plausibly contribute to problematic alcohol use. All other nominally-significant associations identify brain structure as a potential causal contributor to problematic alcohol use; there was no evidence suggesting that problematic alcohol use may cause differences in brain structure. Collectively, these results challenge common interpretations that associations between alcohol use and brain structure reflect consequences of alcohol, instead supporting emerging work suggesting that brain structure may reflect a predispositional risk factor for alcohol involvement.