Electrocardiogram (ECG) signals are often used to diagnose cardiac status. However, most of the existing ECG diagnostic methods only use the time-domain information, resulting in some obviously lesion information in frequency-domain of ECG signals are not being fully utilized. Therefore, we propose a method to fuse the time and frequency domain information in ECG signals by convolutional neural network (CNN). First, we adapt multi-scale wavelet decomposition to filter the ECG signal; Then, R-wave localization is used to segment each individual heartbeat cycle; And then, the frequency domain information of this heartbeat cycle is extracted via fast Fourier transform. Finally, the temporal information is spliced with the frequency domain information and input to the neural network for classification. The experimental results show that the proposed method has the highest recognition accuracy (99.43%) of ECG singles compared with state-of-the-art methods.