In some situations, memory is enhanced by stressful experience, while in others, it is impaired. The specific components of the stress-response that may result in these differing effects remain unclear, and the current study sought to address this knowledge gap. Forty healthy participants (20 women, 20 men) were exposed to emotionally arousing and neutral pictures. Twenty-four hours later, 20 participants underwent a social stressor (speech and math tests), and 20 underwent a control reading task, both followed by a delayed recall task. Cortisol responders to the stress condition (5 men and 1 woman) showed reduced memory retrieval for both neutral and emotionally arousing pictures. Men and women in the stress condition who did not produce a cortisol response showed increased retrieval of unpleasant pictures compared to controls. The results provide further evidence that cortisol is a primary effector in the stress-induced memory retrieval deficit. At the same time, stress can enhance memory retrieval performance, especially for emotional stimuli, when the cortisol response is absent.