Height loss is caused by osteoporosis, vertebral fractures, disc reduction, postural changes, and kyphosis. Marked long-term height loss is reportedly associated with cardiovascular disease and mortality in the elderly. The present study investigated the relationship between short-term height loss and the risk of mortality using the longitudinal cohort data of the Japan Specific Health Checkup Study (J-SHC). Included individuals were aged 40 years or older and received periodic health checkups in 2008 and 2010. The exposure of interest was height loss over the 2 years, and the outcome was all-cause mortality over subsequent follow up. Cox proportional hazard models were used to examine the association between height loss and all-cause mortality. Of the 222,392 individuals (88,285 men, 134,107 women) included in this study, 1436 died during the observation period (mean 4.8 ± 1.1 years). The subjects were divided into two groups based on a cut-off value of height loss of 0.5 cm over 2 years. The adjusted hazard ratio (95% confidence interval) was 1.26 (1.13–1.41) for exposure to height loss ≥ 0.5 cm compared to height loss < 0.5 cm. Height loss ≥ 0.5 cm correlated significantly with an increased risk of mortality compared to height loss < 0.5 cm in both men and women. Even a small decrease in height over 2 years was associated with the risk of all-cause mortality and might be a helpful marker for stratifying mortality risk.