Emotional processing deficits have recently been identified in individuals with traumatic brain injury (TBI), specifically in the domain of facial affect recognition. However, the neural networks underlying these impairments have yet to be identified. In the current study, 42 individuals with moderate to severe TBI and 23 healthy controls performed a task of facial affect recognition (Facial Emotion Identification Test (FEIT)) in order to assess their ability to identify and discriminate six emotions: happiness, sadness, anger, surprise, shame, and fear. These individuals also underwent structural neuroimaging including diffusion tensor imaging to examine white matter (WM) integrity. Correlational analyses were performed to determine where in the brain WM damage was associated with performance on the facial affect recognition task. Reduced performance on the FEIT was associated with reduced WM integrity (fractional anisotropy, mean diffusivity, axial diffusivity, and radial diffusivity) in the inferior longitudinal fasciculus and inferior-fronto-occipital fasciculus in individuals with TBI. Poor performance on the task was additionally associated with reduced gray matter (GM) volume in lingual gyrus and parahippocampal gyrus. The results implicate a pattern of WM and GM damage in TBI that may play a role in emotional processing impairments.