A tendency to look at the left side of faces from the observer’s point of view has been found in older children and adults, but it is not known when this face-specific left gaze bias develops and what factors may influence individual differences in gaze lateralization. Therefore, the aims of this study were to estimate gaze lateralization during face observation and to more broadly estimate lateralization tendencies across a wider set of social and non-social stimuli, in early infancy. In addition, we aimed to estimate the influence of genetic and environmental factors on lateralization of gaze. We studied gaze lateralization in 592 5-month-old twins (282 females, 330 monozygotic twins) by recording their gaze while viewing faces and two other types of stimuli that consisted of either collections of dots (non-social stimuli) or faces interspersed with objects (mixed stimuli). A right gaze bias was found when viewing faces, and this measure was moderately heritable (A = 0.38, 95% CI 0.24; 0.50). A left gaze bias was observed in the non-social condition, while a right gaze bias was found in the mixed condition, suggesting that there is no general left gaze bias at this age. Genetic influence on individual differences in gaze lateralization was only found for the tendency to look at the right versus left side of faces, suggesting genetic specificity of lateralized gaze when viewing faces.