The present research focused on combining Particle Swarm Optimization (PSO) based hybrid deep learning models to classify heart disease images and patient sequences. This study employs Convolutional Neural Networks (CNNs), including VGG 16, VGG 19 and ResNet 50, as well as Recurrent Neu-ral Networks (RNNs), whereby their performance is optimized by PSO to im-prove the accuracy in diagnosing heart disease from CT images together with associated medical history. The models experienced a significant increase in classification performance, using manual hyperparameters tuning by PSO. The combined algorithm PSO with VGG 19 and the RNN model performed best, achieving a precision of 97.78% and becoming the highest recall on testing. The model that we propose uses the modern feature extraction of VGG 19 and an RNN to take into consideration the sequential nature of data, making it very accurate while keeping loss minimal. PSO with VGG 16 and RNN model is also another robust performance with an accuracy of 94.5%.