Morse code radio communication is one of the indispensable shortwave communication methods, but the automatic receiving effect is poor. The accuracy and speed of detection and recognition, in the environment of neighborhood interference and low signal-to-noise ratio (SNR), cannot meet the practical requirements, which is a problem that needs to be solved. Combined with machine learning and deep learning, we propose an automatic Morse signal detection and recognition algorithm based on the ED-FE-CCBC structure. First, based on the time-frequency spectrum, the radio audio signal is adaptively energy sorted, and the target signal is detected according to the characteristics of bandwidth, windowed standard deviation, etc., and its frequency points are calculated. Then, narrow-band digital filtering is performed on the original audio signal with this frequency point as the center frequency, the target signal frequency band is re-extracted on the time-frequency spectrum after noise reduction, and the frequency spectrum is enhanced by pseudo-color. Finally, the pseudo-color image is put into the proposed network for identification and to realize end-to-end decoding. To evaluate the algorithm’s effectiveness, we use two parts of the data of simulation signal and professional training terminal audio to conduct experiments. The results show that in a complex channel environment, the ED-FE-CCBC algorithm can effectively detect and recognize Morse signals. While ensuring real-time performance, the recognition accuracy is better than the most advanced methods.