The immediate extinction deficit describes a higher return of fear when extinction takes place immediately after fear acquisition compared to a delayed extinction design. One explanation for this phenomenon encompasses the remaining emotional arousal evoked by fear acquisition to be still present during immediate, but not delayed extinction. In the present study, the predictive learning task, a learning task not involving arousal or stress, was used testing the hypothesis that no immediate extinction deficit should occur in this neutral task. Twenty-six participants underwent an immediate extinction procedure and were tested in a recall session 24 h later. For the delayed extinction group (n = 26), acquisition, extinction, and recall were realized 24 h apart from each other. Recall performance of a third group (n = 26) was tested 48 h after the immediate extinction procedure. The immediate extinction deficit was indeed observed for a stimulus not subject to a contextual change from acquisition to extinction, but not for other stimuli involving contextual changes or no extinction control stimuli. Even in a neutral learning task and without emotional arousal, the immediate extinction deficit could be detected but was restricted to the specific contextual embedding of stimuli. Thus, contextual processing appears to differentially modulate the emergence of the immediate extinction deficit.