Music and speech are complex and distinct auditory signals which are foundational to the human experience. The mechanisms underpinning each domain are widely investigated. However, how little acoustic information is in fact required to distinguish between them remains an open question. Here we test the hypothesis that a sound’s amplitude modulation (AM) rate is a critical acoustic feature. In contrast to paradigms using ecologically valid, complex acoustic signals (that can be challenging to interpret), we use an aggressively reductionist approach: we artificially generate perceptually ambiguous noise-synthesized audio signals with specific AM parameters. Across four experiments (N = 335), excerpts with a lower peak AM frequency (< 2 Hz) and higher temporal regularity tend to be judged as music; the opposite holds for speech. Furthermore, the effect of peak AM frequency is associated with musical judgements, especially among musically sophisticated participants. The data suggest that the auditory system can rely on a low-level acoustic property as basic as AM to distinguish music from speech, a surprising principle that provokes both neurophysiological and evolutionary experiments and speculations.