Patients with Alzheimer's disease have category-specific semantic memory difficulty for natural relative to manufactured objects. We assessed the basis for this deficit by asking healthy adults and patients to judge whether pairs of words share a feature (e.g. "banana:lemon-COLOR"). In an fMRI study, healthy adults showed gray matter (GM) activation of temporal-occipital cortex (TOC) where visual-perceptual features may be represented, and prefrontal cortex (PFC) which may contribute to feature selection. Tractography revealed dorsal and ventral stream white matter (WM) projections between PFC and TOC. Patients had greater difficulty with natural than manufactured objects. This was associated with greater overlap between diseased GM areas correlated with natural kinds in patients and fMRI activation in healthy adults for natural kinds. The dorsal WM projection between PFC and TOC in patients correlated only with judgments of natural kinds. Patients thus remained dependent on the same neural network as controls during judgments of natural kinds, despite disease in these areas. For manufactured objects, patients' judgments showed limited correlations with PFC and TOC GM areas activated by controls, and did not correlate with the PFC-TOC dorsal WM tract. Regions outside of the PFC-TOC network thus may help support patients' judgments of manufactured objects. We conclude that a large-scale neural network for semantic memory implicates both feature knowledge representations in modality-specific association cortex and heteromodal regions important for accessing this knowledge, and that patients' relative deficit for natural kinds is due in part to their dependence on this network despite disease in these areas.