The notion of a mental time-line (i.e., past corresponds to left and future corresponds to right) supports the conceptual metaphor view assuming that abstract concepts like ''time'' are grounded in cognitively more accessible concepts like ''space.'' In five experiments, we further investigated the relationship between temporal and spatial representations and examined whether or not the spatial correspondents of time are unintentionally activated. We employed a priming paradigm, in which visual or auditory prime words (i.e., temporal adverbs such as yesterday, tomorrow) preceded a colored square. In all experiments, participants discriminated the color of this square by responding with the left or the right hand. Although the temporal reference of the priming adverb was task irrelevant in Experiment 1, visually presented primes facilitated responses to the square in correspondence with the direction of the mental time-line. This priming effect was absent in Experiments 2, 3, and 5, in which the primes were presented auditorily and the temporal reference of the words could be ignored. The effect, however, emerged when attention was oriented to the temporal content of the auditory prime words in Experiment 4. The results suggest that task demands differentially modulate the activation of the mental time-line within the visual and auditory modality and support a flexible association between conceptual codes.