Processing speed is a fundamental aspect of human cognition and intelligence. Many studies from the last decades report that processing speed, typically measured as mean reaction time in simple cognitive tasks, significantly slows down in old age and already declines in young and middle adulthood. Our study employs a Bayesian diffusion model approach to disentangle different cognitive components involved in simple decision-making. We apply our model to a large data set of more than one million participants, which allows us to provide fine-grained and robust analyses of age differences. Since standard Bayesian methods are not suitable to data sets of this size, we use a novel deep learning method for parameter estimation. Our results indicate that processing speed is stable from young adulthood until an age of about 60. The typical age-related slowdown in mean response times in this age range seems attributable to increases in decision caution and slower non-decisional processes – like encoding and motor response – but not to differences in cognitive processing speed. Our research has important implications for all fields concerned with age-related patterns in cognition and challenges widespread beliefs about the relationship between age and cognitive speed.