In computed tomography, coronary artery calcium (CAC) scores are influenced by image reconstruction. The effect of a newly introduced deep learning-based reconstruction (DLR) on CAC scoring in relation to other algorithms is unknown. The aim of this study was to evaluate the effect of four generations of image reconstruction techniques (filtered back projection (FBP), hybrid iterative reconstruction (HIR), model-based iterative reconstruction (MBIR), and DLR) on CAC detectability, quantification, and risk classification. First, CAC detectability was assessed with a dedicated static phantom containing 100 small calcifications varying in size and density. Second, CAC quantification was assessed with a dynamic coronary phantom with velocities equivalent to heart rates of 60–75 bpm. Both phantoms were scanned and reconstructed with four techniques. Last, scans of fifty patients were included and the Agatston calcium score was calculated for all four reconstruction techniques. FBP was used as a reference. In the phantom studies, all reconstruction techniques resulted in less detected small calcifications, up to 22%. No clinically relevant quantification changes occurred with different reconstruction techniques (less than 10%). In the patient study, the cardiovascular risk classification resulted, for all reconstruction techniques, in excellent agreement with the reference (κ = 0.96–0.97). However, MBIR resulted in significantly higher Agatston scores (61 (5.5–435.0) vs. 81.5 (9.25–435.0); p < 0.001) and 6% reclassification rate. In conclusion, HIR and DLR reconstructed scans resulted in similar Agatston scores with excellent agreement and low-risk reclassification rate compared with routine reconstructed scans (FBP). However, caution should be taken with low Agatston scores, as based on phantom study, detectability of small calcifications varies with the used reconstruction algorithm, especially with MBIR and DLR.