Taking another person's perspective provides a means to infer their beliefs and intentions (known as Theory of Mind), which is an essential part of social interaction. In this article, we examined how different subcomponents of perspective-taking change beyond childhood in a large sample (N = 263) of adolescents, young adults, and older adults, and tested the degree to which age-related changes in perspective-taking are mediated by executive functions. Participants completed three tasks that assessed: (a) the likelihood of making social inferences, (b) judgments about an avatar's visual and spatial perspective, and (c) their ability to use an avatar's visual perspective to assign reference in language. Results revealed that while the likelihood of correctly inferring others' mental states increased linearly between adolescence and older adulthood (likely reflecting accumulating social experience over the lifespan), the ability to judge an avatar's perspective and use this to assign reference was subject to developmental changes from adolescence to older age, with performance peaking in young adulthood. Correlation and mediation analyses incorporated three measures of executive functioning (inhibitory control, working memory, and cognitive flexibility) and revealed that executive functions contribute to perspective-taking ability in these tasks (particularly during development), but largely do not mediate the effect of age on perspective-taking. We discuss how these results fit with models of mentalizing that predict distinct patterns of social development depending on the maturation of cognitive and language mechanisms.