Neonates show broad-based, universal speech perception abilities, allowing them to acquire any language. Moreover, an increasing body of research shows that prenatal experience with speech, which is a low-pass signal mainly preserving prosody, already shapes those abilities. In this review, we first provide a summary of the empirical evidence available today on newborns’ universal and experience-modulated speech perception abilities. We then interpret these findings in a new framework, focusing on the role of the prenatal prosodic experience in speech perception development. We argue that the chronological sequence of infants’ experience with speech, starting before birth with a low-pass filtered signal and continuing with the full-band signal after birth, sets up the prosodic hierarchy and a cascade of embedded neural oscillations as its brain correlate, laying the foundations for language acquisition. Prosody, constituting infants’ very first experience with language, may thus play a fundamental role in speech perception and language development.