Human infants would seem to face a daunting challenge in selecting what they should attend, encode and remember. We investigated whether early in life, infants might use others’ attention as an exploitable source of information filtering, by prioritizing the encoding of events that are co-witnessed with someone else over events witnessed alone. In a series of studies (n=255), we show that infants who can otherwise remember an object’s location, misremembered the object where another agent had seen it, even if infants themselves had subsequently seen the object move somewhere else. With further exploratory analyses, we also found that infants’ attention to the agent rather than the object seems to drive their memory for the object’s location. This series points to an initial encoding bias that likely facilitates information selection but which can, under some circumstances, lead to predictable memory errors.