The present study was designed to investigate the effect of a genetic risk factor for Alzheimer's disease (AD), ApolipoproteinE 4 (APOE4), on the ability of the brain to modulate activation in response to cognitive challenge in a lifespan sample of healthy human adults. A community-based sample of 181 cognitively intact, healthy adults were recruited from the Dallas-Fort Worth metroplex. Thirty-one APOE4ϩ individuals (48% women), derived from the parent sample, were matched based on sex, age, and years of education to 31 individuals who were APOE4-negative (APOE4Ϫ). Ages ranged from 20 to 86 years of age. Blood oxygen level-dependent functional magnetic resonance imaging was collected during the performance of a visuospatial distance judgment task with three parametric levels of difficulty. Multiple regression was used in a whole-brain analysis with age, APOE group, and their interaction predicting functional brain modulation in response to difficulty. Results revealed an interaction between age and APOE in a large cluster localized primarily to the bilateral precuneus. APOE4Ϫ individuals exhibited age-invariant modulation in response to task difficulty, whereas APOE4ϩ individuals showed age-related reduction of modulation in response to increasing task difficulty compared with 4Ϫ individuals. Decreased modulation in response to cognitive challenge was associated with reduced task accuracy as well as poorer name-face associative memory performance. Findings suggest that APOE4 is associated with a reduction in the ability of the brain to dynamically modulate in response to cognitive challenge. Coupled with a significant genetic risk factor for AD, changes in modulation may provide additional information toward identifying individuals potentially at risk for cognitive decline associated with preclinical AD.