Gait changes during aging and differs between sexes. Inertial measurement units (IMUs) enable accurate quantitative evaluations of gait in ambulatory environments and in large populations. This study aims to provide IMU-based gait parameters’ values derived from a large longitudinal cohort study in older adults. We measured gait parameters, such as velocity, step length, time, variability, and asymmetry, from straight, self-paced 20-m walks in older adults (four visits: 715/1102/1017/957 participants) every second year over 6 years using an IMU at the lower back. Moreover, we calculated the associations of gait parameters with sex and age. Women showed lower gait speed, step length, step time, stride time, swing time, and stance time, compared to men. Longitudinal analyses suggest that these parameters are at least partly deteriorating within the assessment period of 2 years, especially in men and at an older age. Variability and asymmetry parameters show a less clear sex- and age-associated pattern. Altogether, our large longitudinal dataset provides the first sex-specific information on which parameters are particularly promising for the detection of age-related gait changes that can be extracted from an IMU on the lower back. This information may be helpful for future observational and treatment studies investigating sex and age-related effects on gait, as well as for studies investigating age-related diseases.