Objective: The availability of online electrocardiogram (ECG) repositories can aid researchers in developing automated cardiac abnormality diagnostic systems. Using such ECG repositories, this study aims to develop an algorithm that can assist physicians in diagnosing cardiac abnormalities. Approach: The PhysioNet/CinC 2021 Challenge has opened the venues for creating benchmark algorithms using standard and relatively diverse 12-lead ECG datasets. This work attempts to create a new machine learning approach for identifying common cardiac abnormalities using an ensemble-based classification with two models resulting from two different feature sets. The first feature set extracts RR variability based information by deploying Fourier-Bessel (FB) expansion. The second feature set is composed of time- and frequency-domains-based hand-crafted features. Two long short-term memory (LSTM)-based classifiers are trained using these two feature sets as input to categorize ECG signals. Predictions from these two models are fused to arrive at a final medical decision that improves the multi-label classification of the given ECG signals into twenty-six categories. \\ Main results: We participated in the George B. Moody Physionet Challenge 2021 as team 'Medics', and the proposed methodology was evaluated for all five lead combinations. The challenge scoring metrics obtained on the test data for twelve-, six-, four-, three-, and two-leads combinations are 0.360, 0.368, 0.376, 0.323, and 0.381, respectively. The proposed methodology was ranked 11th among all the follow-up entries of the Challenge. Significance: The obtained results of the proposed method justify the use of an ensemble classifier developed using the extracted feature sets for devising a diagnostic system for detecting and identifying common cardiac problems.