Medical imaging provides valuable input for managing cardiovascular disease (CVD), ranging from risk assessment to diagnosis, therapy planning and follow-up. Artificial intelligence (AI) based medical image analysis algorithms provide nowadays state-of-the-art results in CVD management, mainly due to the increase in computational power and data storage capacities. Various challenges remain to be addressed to speed-up the adoption of AI based solutions in routine CVD management. Although medical imaging and in general health data are abundant, the access and transfer of such data is difficult to realize due to ethical considerations. Hence, AI algorithms are often trained on relatively small datasets, thus limiting their robustness, and potentially leading to biased or skewed results for certain patient or pathology sub-groups. Furthermore, explainability and interpretability have become core requirements for AI algorithms, to ensure that the rationale behind output inference can be revealed. The paper focuses on recent developments related to these two challenges, discusses the clinical impact of proposed solutions, and provides conclusions for further research and development. It also presents examples related to the diagnosis of stable coronary artery disease, a whole-body circulation model for the assessment of structural heart disease, and to the diagnosis and treatment planning of aortic coarctation, a congenital heart disease.