Previous studies suggest that task-irrelevant changing-state sound interferes specifically with the processing of serial order information in the focal task (e.g., serial recall from short-term memory), whereas a deviant sound in the auditory background is supposed to divert central attention, thus producing distraction in various types of cognitive tasks. Much of the evidence for this distinction rests on the observed dissociations in auditory distraction between serial and non-serial short-term memory tasks. In this study, both the changing-state effect and the deviation effect were contrasted between serial digit recall and mental arithmetic tasks. In three experiments (two conducted online), changing-state sound was found to disrupt serial recall, but it did not lead to a general decrement in performance in different mental arithmetic tasks. In contrast, a deviant voice in the stream of irrelevant speech sounds did not cause reliable distraction in serial recall and simple addition/subtraction tasks, but it did disrupt a more demanding mental arithmetic task. Specifically, the evaluation of math equations (multiplication and addition/subtraction), which was combined with a pair-associate memory task to increase the task demand, was found to be susceptible to auditory distraction in participants who did not serially rehearse the pair-associates. Together, the results support the assumption that the interference produced by changing-state sound is highly specific to tasks that require serial-order processing, whereas auditory deviants may cause attentional capture primarily in highly demanding cognitive tasks (e.g., mental arithmetic) that cannot be solved through serial rehearsal.