In the dual-route language model, the dorsal pathway is known for sound-to-motor mapping, but the role of the ventral stream is controversial. With the goal of enhancing our understanding of language models, this study investigated the diffusion characteristics of candidate tracts in aphasic patients. We evaluated 14 subacute aphasic patients post-stroke and 11 healthy controls with language assessment and diffusion magnetic resonance imaging. Voxel-based lesion-symptom mapping found multiple linguistic associations for the ventral stream, while automated fiber quantification (AFQ) showed, via reduced fractional anisotropy (FA) and axial diffusivity with increased radial diffusivity (all corrected p < 0.05), that the integrity of both the left dorsal and ventral streams was compromised. The average diffusion metrics of each fascicle provided by AFQ also confirmed that voxels with significant FA-language correlations were located in the ventral tracts, including the left inferior fronto-occipital fascicle (IFOF) (comprehension: r = 0.839, p = 0.001; repetition: r = 0.845, p = 0.001; naming: r = 0.813, p = 0.002; aphasia quotient: r = 0.847, p = 0.001) and uncinate fascicle (naming: r = 0.948, p = 0.001). Furthermore, point-wise AFQ revealed that the segment of the left IFOF with the strongest correlations was its narrow stem. The temporal segment of the left inferior longitudinal fascicle was also found to correlate significantly with comprehension (r = 0.663, p = 0.03) and repetition (r = 0.742, p = 0.009). This preliminary study suggests that white matter integrity analysis of the ventral stream may have the potential to reveal aphasic severity and guide individualized rehabilitation. The left IFOF, specifically its narrow stem segment, associates with multiple aspects of language, indicating an important role in semantic processing and multimodal linguistic functions.