In this article, we used High Angular Resolution Diffusion Imaging (HARDI) with advanced anatomically constrained particle filtering tractography to investigate the role of the arcuate fasciculus (AF) and the middle longitudinal fasciculus (MdLF) in speech perception in noise in younger and older adults. Fourteen young and 15 elderly adults completed a syllable discrimination task in the presence of broadband masking noise. Mediation analyses revealed few effects of age on white matter (WM) in these fascicles but broad effects of WM on speech perception, independently of age, especially in terms of sensitivity and criterion (response bias), after controlling for individual differences in hearing sensitivity and head size. Indirect (mediated) effects of age on speech perception through WM microstructure were also found, after controlling for individual differences in hearing sensitivity and head size, with AF microstructure related to sensitivity, response bias and phonological priming, and MdLF microstructure more strongly related to response bias. These findings suggest that pathways of the perisylvian region contribute to speech processing abilities, with relatively distinct contributions for the AF (sensitivity) and MdLF (response bias), indicative of a complex contribution of both phonological and cognitive processes to age‐related speech perception decline. These results provide new and important insights into the roles of these pathways as well as the factors that may contribute to elderly speech perception deficits. They also highlight the need for a greater focus to be placed on studying the role of WM microstructure to understand cognitive aging.