Enhanced visual cortex activation by negative compared to neutral stimuli is often attributed to modulating feedback from the amygdala, but evidence from lesion studies is scarce, particularly regarding differential effects of left and right amygdala lesions.Therefore, we compared visual cortex activation by negative and neutral complex scenes in an event-related fMRI study between 40 patients with unilateral temporal lobe resection (TLR; 19 left [lTLR], 21 right [rTLR]), including the amygdala, and 20 healthy controls. We found preserved hemodynamic emotion modulation of visual cortex in rTLR patients and only subtle reductions in lTLR patients. In contrast, rTLR patients showed a significant decrease in visual cortex activation irrespective of picture content. In line with this, healthy controls showed small emotional modulation of the left amygdala only, while their right amygdala was activated equally by negative and neutral pictures. Correlations of activation in amygdala and visual cortex were observed for both negative and neutral pictures in the controls. In both patient groups, this relationship was attenuated ipsilateral to the TLR. Our results support the notion of reentrant mechanisms between amygdala and visual cortex and suggest laterality differences in their emotion-specificity. While right medial temporal lobe structures including the amygdala seem to influence visual processing in general, the left medial temporal lobe appears to contribute specifically to emotion processing. Still, effects of left TLR on visual emotion processing were relatively subtle. Therefore, hemodynamic correlates of visual emotion processing are likely supported by a distributed cerebral network, challenging an amygdalocentric view of emotion processing.