With advancing age, the distinctiveness of neural representations of information declines. While the finding of this so-called "age-related neural dedifferentiation" in category-selective neural regions is well-described, the contribution of age-related changes in network organization to dedifferentiation is unknown. Here, we asked whether age differences in a) functional connectivity to category-selective neural regions and b) segregation of the visual network (i.e., network dedifferentiation) contribute to regional dedifferentiation of categorical representations. Younger and older adults viewed blocks of face and house stimuli in the fMRI scanner. We found an age-related decline in neural distinctiveness for faces in the fusiform gyrus (FG) and for houses in the parahippocampal gyrus (PHG). Functional connectivity analyses revealed age differences in connectivity between the FG and visual network as well as age-related dedifferentiation of the visual network. Interindividual correlations demonstrated that regional distinctiveness was related to connectivity of category-selective regions to the visual network as well as network segregation. Thus, dedifferentiation of categorical representations may be linked to age-related reorganization of functional networks.