Emotional stimuli can disrupt or enhance task performance according to factors that are presently poorly understood. One potentially important determinant is the sensory modality involved. In unimodal visual paradigms (visual task-irrelevant stimuli during a visual task) emotional stimuli frequently produce distraction effects; however, the effects across modalities appear more complex and may also depend on factors related to stimulus timing. It is entirely unclear how task-irrelevant visual stimuli impact auditory task performance in cross-modal paradigms. This project explored task performance as a function of sensory modality, emotional valence, and stimulus timing. In Study 1, participants (N = 50) completed a visual stimulus detection task in the presence of task-irrelevant negative and neutral images and sounds. Accuracy was disrupted in the presence of visual but not auditory emotional stimuli, particularly when the target and task-irrelevant stimulus appeared simultaneously. In Study 2, participants (N = 38) completed an equivalent auditory stimulus detection task. In sharp contrast to the effects observed with visual targets, response times and accuracy were enhanced in the presence of auditory emotional stimuli at the first timepoint but disrupted at later timepoints. However, there was no effect of task-irrelevant visual stimuli on auditory task performance. These findings demonstrate the importance of both sensory modality and timing in determining how emotional stimuli affect task performance and lay the groundwork for future studies examining the interaction between emotional and attentional processes.