Well before they understand their first words, infants have begun to link language and cognition. This link is initially broad: At 3 months, listening to both human and nonhuman primate vocalizations supports infants’ object categorization, a building block of cognition. But by 6 months, the link has narrowed: Only human vocalizations support categorization. What mechanisms underlie this rapid tuning process? Here, we document the crucial role of infants’ experience as infants tune this link to cognition. Merely exposing infants to nonhuman primate vocalizations permits them to preserve, rather than sever, the link between these signals and categorization. Exposing infants to backward speech—a signal that fails to support categorization in the first year of life—does not have this advantage. This new evidence illuminates the central role of early experience as infants specify which signals, from an initially broad set, they will continue to link to core cognitive capacities.