Cardiovascular diseases (CVDs) are the leading cause of death globally. Detecting this kind of disease represents the principal concern of many scientists, and techniques belonging to various fields have been developed to attain accurate predictions. The aim of the paper is to investigate the potential of the classical, evolutionary, and deep learning-based methods to diagnose CVDs and to introduce a couple of complex hybrid techniques that combine hyper-parameter optimization algorithms with two of the most successful classification procedures: support vector machines (SVMs) and Long Short-Term Memory (LSTM) neural networks. The resulting algorithms were tested on two public datasets: the data recorded by the Cleveland Clinic Foundation for Heart Disease together with its extension Statlog, two of the most significant medical databases used in automated prediction. A long series of simulations were performed to assess the accuracy of the analyzed methods. In our experiments, we used F1 score and MSE (mean squared error) to compare the performance of the algorithms. The experimentally established results together with theoretical consideration prove that the proposed methods outperform both the standard ones and the considered statistical methods. We have developed improvements to the best-performing algorithms that further increase the quality of their results, being a useful tool for assisting the professionals in diagnosing CVDs in early stages.