Electrocardiogram (ECG) signal is informative as well as non-invasive clinical tool to diagnose cardiac diseases of human heart. However, the diagnosis requires professionals’ clarification and is also time-consuming. To make the diagnosis proficient, a novel convolutional neural network (CNN) has been proposed for automatic arrhythmia detection. In this work, the ECG data collected from the MIT-BIH database have been preprocessed, and segmented in short ECG segments of 60 s. Then, all these segments have been transformed into scalogram images obtained from time-frequency analysis using continuous wavelet transform (CWT). Finally, these scalogram images have been exploited as an input for our designed CNN classifier to classify cardiac arrhythmia. In this approach, the overall accuracy, sensitivity, and specificity are 99.39%, 98.79%, and 100% respectively. Proposed CNN model has significant advantages, and it can be used to differentiate the healthy and arrhythmic patients effectively.