Background: Vitamin D supplementation may be associated with lower cardiovascular (CV) events, but the data are controversial. It remains speculative whether vitamin D supplementation has a direct effect on coronary atherosclerosis. We therefore set out to assess the influence of vitamin D supplementation on the coronary atherosclerosis profile quantified by coronary computed tomography angiography (CTA) in a retrospective case–control cohort study. Methods: 176 patients (age: 62.4 ± 10.4) referred to coronary CTA for clinical indications were included. A total of 88 patients receiving vitamin D supplementation (mean duration 65.3 ± 81 months) were 1:1 propensity score matched with 88 controls for age, gender, smoking, arterial hypertension, positive family history, dyslipidemia, and diabetes. Coronary stenosis severity (CAD-RADSTM), mixed plaque burden (weighted for non-calcified), high-risk-plaque (HRP) features, and plaque density (HU) were quantified by CTA. Serum 25-hydroxyvitamin D (OH)-levels were measured in 138 patients and categorized into four groups (0: <20 ng/mL; 1: 20–40 ng/mL; 2: 40–60 ng/mL; and 3: >60 ng/mL) and compared with CTA. Results: The prevalence of atherosclerosis by CTA was similar in both groups (75.6% versus 74.3%, p = 0.999), >50% coronary stenosis was slightly higher in controls (p = 0.046), but stenosis severity score (CAD-RADS) was not different (p = 0.106). Mixed plaque burden (weighted for non-calcified) was lower in patients receiving vitamin D supplementation (p = 0.002) and high-risk-plaque prevalence was markedly lower (3.8% versus 32%, p < 0.001). CT plaque density (HU) was higher (p < 0.001) in the vitamin D group. Patients with serum vitamin D (OH) levels >60 ng/mL had higher plaque density (p = 0.04), indicating more calcified and less vulnerable plaque. Conclusions: In this retrospective case–control cohort study, vitamin D supplementation was associated with less high-risk plaque, less non-calcified plaque burden, and a higher calcified plaque independent of CV risk factors.