Bilingual infants acquire languages in a variety of language environments. Some caregivers follow a one-person-one-language approach in an attempt to not ‘confuse’ their child. However, the central assumption that infants can keep track of what language a person speaks has not been tested. In two studies, we tested whether bilingual and monolingual 5-, 12- and 18-month-olds spontaneously form language-person associations. In both studies, infants were familiarized to a man and a woman, each speaking a different language, and tested on trials where they either spoke the same language or switched to a different language. In Study 1, infants only heard the speaker, and in Study 2 infants saw and heard the speaker. Bilinguals and monolinguals did not look longer to Switch compared to Same trials; thus, there was no evidence that infants form person-language associations spontaneously. This indicates that at least early in development, a central assumption of the one-person-one-language approach is not supported.