Mental imagery is involved in a wide variety of cognitive abilities, including reasoning, spatial navigation, and memory. Cognitive aging is associated with impairments in these abilities, suggesting diminished fidelity of mental images in older adults may be related to diverse cognitive deficits. However, an age-related deficit in mental imagery and its role in memory impairment is still a matter of debate. Previous human fMRI studies demonstrated that visual imagery activates representations in category-selective visual cortex via top-down control mechanisms. Here, we use fMRI to show that normal aging is associated with diminished selectivity of visual cortex activation during visual imagery, with a corresponding reduction in the selectivity of functional connections between prefrontal cortex and visual cortices. Moreover, a relationship between reduced imagery selectivity and visual memory in older adults was established. These results reveal that aging disrupts neural networks that subserve mental imagery and offers evidence of this as a factor in age-related memory decline.