Little is known regarding the association between physical fitness and bone health in older Korean men. This study investigated the relationship between estimated cardiorespiratory fitness (eCRF) and bone mineral density (BMD). This cross-sectional study included 2715 Korean men aged 50 years and older selected from those who participated in the 2008–2011 Korea National Health and Nutritional Examination and Survey. eCRF was obtained using a sex-specific algorithm developed on the basis of age, body mass index, resting heart rate, and physical activity and classified into low, middle, and high categories. Femoral neck BMD was assessed by dual X-ray absorptiometry. Odds ratios (OR) and 95% confidence intervals (CI) for osteopenia, osteoporosis, and low BMD were calculated for eCRF categories in models fully adjusted for age, waist circumference, education, income, smoking, heavy alcohol intake, serum vitamin D, serum parathyroid hormone, and dietary intake of energy, protein, calcium, and vitamins A and C. Overall, eCRF levels were positively associated with BMD and negatively with prevalence of osteopenia, osteoporosis, and low BMD. Logistic regression showed inverse trends in the risks of osteopenia (high vs. low: OR = 0.692; 95% CI, 0.328–0.517; p = 0.049) and low BMD (high vs. low: OR = 0.669; 95% CI, 0.497–0.966; p = 0.029) by eCRF category in models fully adjusted for all the measured covariates. The current findings suggest that maintaining high eCRF via regular physical activity may contribute to attenuation of age-related loss of BMD and decreased risk for low BMD in older Korean men.