Infant emotional stimuli can preferentially engage adults’ attention and provide valuable information essential for successful interaction between adults and infants. Exploring the neural processes of recognizing infant stimuli promotes better understandings of the mother-infant attachment mechanisms. Here, combining task-functional magnetic resonance imaging (Task-fMRI) and resting-state fMRI (rs-fMRI), we investigated the effects of infants’ faces on the brain activity of adults. Two groups including 26 women and 25 men were recruited to participate in the current study. During the task-fMRI, subjects were exposed to images of infant emotional faces (including happy, neutral, and sad) randomly. We found that the brains of women and men reacted differently to infants’ faces, and these differential areas are in facial processing, attention, and empathetic networks. The rs-fMRI further showed that the connectivity of the default-mode network-related regions increased in women than in men. Additionally, brain activations in regions related to emotional networks were associated with the empathetic abilities of women. These differences in women might facilitate them to more effective and quick adjustments in behaviors and emotions during the nurturing infant period. The findings provide special implications and insights for understanding the neural processing of reacting to infant cues in adults.