Life expectancy has steadily increased for over a century; we thus live longer and are more likely to experience cognitive difficulties such as increased distractibility which can hamper autonomy. This cross-sectional behavioral study aimed to characterize the decline of the cognitive components of distractibility during typical aging, and the onset of this decline. 191 participants from 21 to 86 years old, distributed within seven age groups, were tested using the Competitive Attention Test. Results indicate that cognitive components contributing to distractibility follow different trajectories with aging: voluntary orienting remains stable from 21 to 86 years old, sustained attention decreases while distraction increases between 26 and 86 years old, finally, impulsivity is lower in older compared to younger adults. Increased distractibility in older adults thus seems to result from a dominance of involuntary over voluntary attention processes, whose detrimental effect on performance is partly compensated by enhanced recruitment of motor control.