Working memory (WM) declines with increasing age. The WM capacity is often measured by means of the computerized version of the n-back task. Although the n-back task is widely used in aging research, little is known about its construct validity and specific cognitive functions involved in this task. Moreover, to date, no studies analyzed the construct validity as a function of age. To this end, we conducted a study in a sample of N = 533 individuals aged between 20 and 80 years. The sample was divided into three age groups: young (20–40), middle-aged (41–60), and old (61–80 years). A number of psychometric tests was selected that measure attention, memory, and executive control to elucidate the impact of these constructs on n-back performance. A series of correlation analyses was conducted to assess the relationship between n-back performance and specific cognitive functions in each age group separately. The results show a progressive increase in reaction times and a decrease in the proportion of detected targets from young to old subjects. Age-related impairments were also found in all psychometric tests except for the vocabulary choice test measuring crystallized intelligence. Most importantly, correlations yielded different age-related patterns of functions contributing to performance in the n-back task: whereas performance was most related to executive functions in young age, a combination of attentional and executive processes was associated with performance in middle-aged subjects. In contrast, in older age, mainly attentional, verbal memory, and updating and to a lesser extent executive processes seem to play a crucial role in the n-back task, suggesting a shift of processing strategies across the lifespan.