Six experiments based on visual preference procedures were conducted to examine gender categorization of female versus male faces by infants aged 3 to 4 months. In experiment 1, infants familiarized with male faces preferred a female face over a novel male face, but infants familiarized with female faces divided their attention between a male face and a novel female face. Experiment 2 demonstrated that these asymmetrical categorization results were likely due to a spontaneous preference for females. Experiments 3 and 4 showed that the preference for females was based on processing of the internal facial features in their upright orientation, and not the result of external hair cues or higher-contrast internal facial features. While experiments 1 through 4 were conducted with infants reared with female primary caregivers, experiment 5 provided evidence that infants reared with male primary caregivers tend to show a spontaneous preference for males. Experiment 6 showed that infants reared with female primary caregivers displayed recognition memory for individual females, but not males. These results suggest that representation of information about human faces by young infants may be influenced by the gender of the primary caregiver.