Early language development is characterized by large individual variation. Several factors were proposed to contribute to individual pathways of language acquisition in infancy and childhood. One of the biologically based explaining factors is temperament, however, the exact contributions and the timing of the effects merits further research. Pre-term status, infant sex, and environmental factors such as maternal education and maternal language are also involved. Our study aimed to investigate the longitudinal relationship between infant temperament and early language development, also considering infant gender, gestational age, and birthweight. Early temperament was assessed at 6, 9, 18, 24, and 30 months with the Very Short Form of Infant Behavior Questionnaire (IBQ-R) and the Very Short Form of Early Childhood Behavior Questionnaire (ECBQ). Early nonverbal communication skills, receptive and expressive vocabulary were evaluated with the Hungarian version of The MacArthur Communicative Development Inventory (HCDI). Our study adds further evidence to the contribution of infant temperament to early language development. Temperament, infant gender, and gestational age were associated with language development in infancy. Infants and toddlers with higher Surgency might enter communicative situations more readily and show more engagement with adult social partners, which is favorable for communication development. Gestational age was previously identified as a predictor for language in preterm infants. Our results extend this association to the later and narrower gestational age time window of term deliveries. Infants born after longer gestation develop better expressive vocabulary in toddlerhood. Gestational age may mark prenatal developmental processes that may exert influence on the development of verbal communication at later ages.