The role cardiac function plays in the predilection for, and progression of, Alzheimer's disease (AD), is complex and unclear. While a number of low-frequency genetic variants of large effect-size have been shown to underlie both cardiovascular disease and dementia, recent studies have highlighted the importance of common genetic variants of small-effect size, which, in aggregate, are embodied by a polygenic risk score (PRS). In this study we aim to investigate the effect of polygenic risk for coronary artery disease (CAD) on brain atrophy in AD using whole brain volume (WBV) and put our findings in context with the polygenic risk for AD and presumed small vessel disease as quantified by white matter hyperintensities (WMH). We used 730 subjects from the ADNI database to investigate PRS effects (beyond APOE) on whole brain volumes, total and regional WMH and amyloid beta across diagnostic groups. In a subset of these subjects (N=602) we utilise longitudinal changes in whole brain volume over a maximum of 24 months using the boundary shift integral approach. Linear regression and linear mixed effects models were used to investigate the effect of WMH at baseline as well as AD-PRS and CAD-PRS on whole brain atrophy and whole brain atrophy acceleration, respectively. All genetic associations were examined under oligogenic (p=1e-5) and the more variant-inclusive polygenic (p=0.5) scenarios. Our results suggest no evidence for a link between PRS score and markers of AD pathology at baseline (when stratified by diagnostic group). However, both AD-PRS and CAD-PRS were associated with longitudinal decline in WBV (AD PRS t=3.3, PFDR=0.007 over 24 months in healthy controls) and surprisingly, under certain conditions WBV atrophy is statistically more correlated with cardiac PRS than AD PRS (CAD PRS t=2.1, PFDR=0.04 over 24 months in the MCI group). Further, in our regional analysis of WMH, AD PRS beyond APOE is predictive of white matter volume in the occipital lobe in AD subjects in the polygenic regime. Finally, the rate of change of brain volume (or atrophy acceleration) may be sensitive to AD polygenic risk beyond APOE in healthy individuals (t=2, p=0.04). For subjects with mild cognitive impairment (MCI), beyond APOE, a more inclusive polygenic risk score including more variants, shows CAD PRS to be more predictive of WBV atrophy, than an oligogenic approach including fewer larger effect size variants.