This article focuses on automatic modulation classification (AMC) in wireless communication systems. A convolutional neural network (CNN) with three layers is introduced for the AMC process. Over degraded channels, it is assumed that the constellation diagrams of received signals do not show sharp points as in the case of pure signals. Instead, the points spread to constitute circle‐shaped objects. With more deterioration in channel conditions, these circle‐shaped objects begin to show overlapping. This behavior motivates us to use object detection, when dealing with the modulation classification task. The selection of the adopted transforms in this article is made from the object detection perspective. Different 2D transforms are considered on the constellation diagrams and compared for better classification performance. These transforms are the Radon transform (RT), the curvelet transform, and the phase congruency (PC). They are applied on the 2D constellation diagrams prior to the classification task with the CNN. The classification of the modulation format at different signal‐to‐noise ratios (SNRs) is considered in this article from the constellation diagrams, and the preprocessed constellation diagrams using RT, curvelet transform, and PC. Seven types of modulation formats are considered in this study to represent both spread and dense constellation diagram patterns, and the study extends from −10 to 10 dB. Analysis of the results indicating the most suitable preprocessor according to the constellation type and the SNR involved is provided.