As cardiovascular diseases have been one of the leading causes of death, early and accurate diagnosis of cardiac abnormalities with less cost becomes particularly important. Given the electrocardiogram (ECG) datasets from multiple sources, there exist many challenges to develop the generalized models that can identify multiple types of cardiac abnormalities from both 12-lead ECG signals and reduced-lead ECG signals. In this study, our objective is to build robust models which can accurately classify 30 types of abnormalities from various lead combinations of ECG signals. Given the challenges of this problem, we proposed a framework for building robust models for ECG signal classification. Firstly, a pre-processing workflow was adopted for each ECG dataset to mitigate the problem of data divergence. Secondly, to capture the lead-wise relations, we used a squeeze-and-excitation deep residual network (SE_ResNet) as our base model. Thirdly, we proposed the cross relabeling strategy and applied the sign-augmented loss function to tackle the corrupted labels in the data. Furthermore, we utilized a pos-if-any-pos ensemble strategy and a dataset-wise cross evaluation strategy to handle the uncertainty of the data distribution in the application. In the Physionet/Computing in Cardiology Challenge 2021, our approach achieved the challenge metric scores of 0.57, 0.59, 0.59, 0.58, 0.57 on 12, 6, 4, 3, 2 lead versions and an averaged challenge metric score of 0.58 over all the lead versions.Using the proposed framework, we developed the models from several large datasets with sufficiently labeled abnormalities. Our models could identify 30 ECG abnormalities accurately based on various lead combinations of ECG signals. The performance on hidden test data demonstrated the effectiveness of the proposed approaches.