Left atrium (LA) size is associated with adverse cardiovascular events. The purpose of this study was to investigate the association of LA enlargement measured by non-contrast CT (NCCT) with traditional cardiovascular risk factors. Individuals aged 60–75 years from the population-based multicentre Danish Cardiovascular Screening (DANCAVAS) trial were included in this cross-sectional study. The LA was manually traced on the NCCT scans, and the largest cross-section area was indexed to body surface area. All traditional risk factors were recorded, and a subgroup received an echocardiographic examination. We enrolled 14,987 individuals. Participants with known cardiovascular disease or lacking measurements of LA size or body surface area were excluded, resulting in 10,902 men for the main analysis and 616 women for a sensitivity analysis. Adjusted multivariable analysis showed a significantly increased indexed LA size by increasing age and pulse pressure, while smoking, HbA1c, and total cholesterol were associated with decreased indexed LA size. The findings were confirmed in a supplementary analysis including left ventricle ejection fraction and mass. In this population-based cohort of elderly men, an association was found between age and pulse pressure and increasing LA size. Surprisingly, smoking, HbA1c, and total cholesterol were associated with a decrease in LA size. This indicates that the pathophysiology behind atrial cardiomyopathy is not only reflected by enlargement, but also shrinking.