The Whorfian hypothesis has received support from recent findings in psychology, linguistics, and anthropology. This evidence has been interpreted as supporting the view that language modulates all stages of perception and cognition, in accordance with Whorf's original proposal. In light of a much broader body of evidence on time perception, I propose to evaluate these findings with respect to their scope. When assessed collectively, the entire body of evidence on time perception shows that the Whorfian hypothesis has a limited scope and that it does not affect early stages of time perception. In particular, all the available evidence shows that the scope of language modulation is limited in the case of time perception, and that the most important mechanisms for time perception are cognitive clocks and simultaneity windows, which we use to perceive the temporal properties of events. Language modulation has distorting effects, but only at later stages of processing or with respect to specific categorization tasks. The paper explains what is the role of these effects in the context of all the available evidence on time cognition and perception.