Language function in the brain, once thought to be highly localized, is now appreciated as relying on a connected but distributed network. The semantic system is of particular interest in the language domain because of its hypothesized integration of information across multiple cortical regions. Previous work in healthy individuals has focused on group-level functional connectivity (FC) analyses of the semantic system, which may obscure interindividual differences driving variance in performance. These studies also overlook the contributions of white matter networks to semantic function. Here, we identified semantic network nodes with a semantic decision fMRI task in 53 typically-aging adults, characterized network organization using structural connectivity (SC), and quantified the segregation and integration of the network using FC. Hub regions were identified in left inferior frontal gyrus. The individualized semantic network was composed of three interacting modules: 1) default-mode module characterized by bilateral medial prefrontal and posterior cingulate regions and also including right-hemisphere homotopes of language regions; 2) left frontal module extending dorsally from inferior frontal gyrus to pre-motor area; and 3) left temporoparietal module extending from temporal pole to inferior parietal lobule. FC within Module3 and integration of the entire network related to a semantic verbal fluency task, but not a matched phonological task. These results support and extend the tri-network semantic model (Xu et al., 2017) and the controlled semantic cognition model (Chiou et al., 2018) of semantic function.