Temporal and spatial representations are not independent of each other. Two conflicting theories provide alternative hypotheses concerning the specific interrelations between temporal and spatial representations. The asymmetry hypothesis (based on the conceptual metaphor theory, Lakoff and Johnson, 1980) predicts that temporal and spatial representations are asymmetrically interrelated such that spatial representations have a stronger impact on temporal representations than vice versa. In contrast, the symmetry hypothesis (based on a theory of magnitude, Walsh, 2003) predicts that temporal and spatial representations are symmetrically interrelated. Both theoretical approaches have received empirical support. From an embodied cognition perspective, we argue that taking sensorimotor processes into account may be a promising steppingstone to explain the contradictory findings. Notably, different modalities are differently sensitive to the processing of time and space. For instance, auditory information processing is more sensitive to temporal than spatial information, whereas visual information processing is more sensitive to spatial than temporal information. Consequently, we hypothesized that different sensorimotor tasks addressing different modalities may account for the contradictory findings. To test this, we critically reviewed relevant literature to examine which modalities were addressed in time-space mapping studies. Results indicate that the majority of the studies supporting the asymmetry hypothesis applied visual tasks for both temporal and spatial representations. Studies supporting the symmetry hypothesis applied mainly auditory tasks for the temporal domain, but visual tasks for the spatial domain. We conclude that the use of different tasks addressing different modalities may be the primary reason for (a)symmetric effects of space on time, instead of a genuine (a)symmetric mapping.