MRI measurements of heart volumes and vessel dimensions typically require specialised cardiac MRI examinations, which can be time consuming and resource intensive. Simpler scans covering the heart and major blood vessels of the abdomen may potentially provide useful clinical information. We used deep learning to segment cardiovascular structures from UK Biobank (UKBB) abdominal MRI scans and extracted six image-derived phenotypes (heart volume, and cross-sectional areas of the aorta and vena cava at key anatomical locations) from 44,541 participants. We found strong correlations between our measurements and those acquired using conventional cardiac MRI (CMR), with the strength of the association reflecting the degree of anatomical similarity between the measurements. We replicated previous findings related to gender differences and age-related changes in heart and vessel dimensions, and identified a significant association between infrarenal descending aorta cross-sectional area and incident aneurysm, and between heart volume and several cardiovascular disorders. In a genome-wide association study we identified 72 associations at 59 loci, of which 15 had not been previously associated with CMR IDPs. Furthermore, we demonstrated substantial genetic correlation with cardiovascular traits including varicose veins, aneurysms, dysrhythmia and cardiac failure. Our heritability enrichment implicated vascular tissue in the heritability of these traits. We derived a polygenic risk score for each trait and demonstrated an association with aneurysm diagnosis, pointing to a potential screening method for individuals at high-risk of this condition. Our work highlights the rich information content of abdominal MRI and the high degree of overlap with CMR.