This study investigated whether the timing of infant motor development is associated with self‐reported and accelerometer‐measured physical activity (PA) and sedentary time (ST) in midlife. This population‐based study consisted of 4098 people born in 1966 in Northern Finland (NFBC 1966). Data on nine infant motor developmental milestones included making sounds, holding up the head, grabbing objects, turning from back to tummy, sitting without support, standing with support, walking with support, standing without support, and walking without support. At the age of 46, PA at leisure time and sitting time was self‐reported. PA and ST were also measured with a wrist‐worn Polar Active accelerometer that was instructed to be worn on the non‐dominant hand 24 h/d for 14 days. A multiple linear regression analysis was used to analyze the association between infant motor development and PA and ST in midlife. Later infant motor development was weakly associated with higher accelerometer‐measured light PA, but not with moderate‐to‐vigorous PA. Later infant locomotor development was associated with lower accelerometer‐measured ST (β −0.07, p = 0.012) and lower self‐reported sitting time at work (β −0.06, p = 0.004) in women. In conclusion, later infant motor development was associated with higher light PA and lower sedentary time at middle age. PA is a multifactorial behavior influenced by various factors from early childhood to midlife. Further research is required before more general conclusions can be drawn.