Background: Obesity in older adults is increasing. The two types of fat involved in obesity are subcutaneous and visceral fat. We investigated the relationship between cardiovascular risk factors, health-related behaviors, and physical fitness based on the level of visceral fat in older men. Methods: Data from 1219 men aged ≥65 years who underwent health checkups at Korean hospitals from 2012 to 2020 were analyzed. Computed tomography (CT) data was used to divide participants into low visceral fat area (LVFA) and high visceral fat area (HVFA) groups. Cardiovascular risk factors such as blood pressure, triglyceride level, high-density cholesterol, and fasting blood glucose were measured. Bio-impedance analysis was used for body composition, and arteriosclerosis was tested by brachial artery pulse wave velocity (baPWV). Physical activity, exercise type, smoking, and alcohol consumption were investigated and cardiorespiratory fitness, grip strength, leg strength, flexibility, balance, and agility were determined. The odds ratio (OR) of HVFA was calculated by logistic regression analysis. Results: HVFA and LVFA groups exhibited significant differences in waist circumference, systolic blood pressure, triglyceride, glucose, baPWV, weekly physical activity, aerobic exercise, and alcohol consumption. Compared to the high fitness group, the low fitness group had an increased OR for HVFA; cardiorespiratory fitness (OR 3.563), grip strength (OR 2.365), leg strength (OR 4.145), and flexibility (OR 2.523). The group with low aerobic and strength training frequency increased OR by 2.5 and 3.2 times compared to the low frequency group, respectively. The OR of HVFA increased 2.8 times in the group which consumed alcohol more than 4 days a week compared to the group with consumption less than monthly. Conclusions: Men with HVFA exhibited lower levels of physical activity, more alcohol consumption, and lower levels of fitness than those with LVFA. Moreover, low levels of fitness, physical activity, and high alcohol consumption increased the risk of HVFA.