Metaphorical association between vertical space and emotional valence is activated by bodily movement toward the corresponding space. Upward or downward manual movement “following” observation of emotional images is reported to alter the perceived valence as more positive or negative. This study aimed to clarify this retrospective emotional modulation. Experiment 1 investigated the effects of temporal order of emotional stimuli and manual movements. Participants performed upward, downward, or horizontal manual movements immediately before or after observation of emotional images; they then rated the valence of the image. The images were rated as more negative in downward- than in horizontal-movement conditions only when the movements followed the image observation. Upward movement showed no effect. Experiment 2 examined the effects of temporal proximity between images, movements, and ratings. The results showed that a 2-s interval either between image and movement or movement and rating nullified the retrospective effect. Bodily movement that corresponds to space–valence metaphor retrospectively, but not prospectively, alters the perceived valence of emotional stimuli. This effect requires temporal proximity between emotional stimulus, the subsequent movement, and rating of the stimulus. With respect to the lack of effect of upward–positive correspondence, anisotropy in effects of movement direction is discussed.