With the miniaturization of communication devices, the number of distributed electromagnetic devices is increasing. In order to achieve effective management of the electromagnetic spectrum, prediction and anomaly detection of the spectrum has become increasingly critical. This paper proposes an algorithmic framework for detecting spectrum anomalies using deep learning techniques. More specifically, the framework includes spectrum prediction and anomaly detection. We use the sliding window method to divide the time series, construct multi-timescale historical data, and train the model with normal data to have high accuracy spectrum prediction capability. We analyze and determine the discriminant function to distinguish the spectral anomalies by calculating the differences between the predicted and real data. The experimental results show that the proposed method outperforms existing baseline algorithms based on real-world spectrum measurement data and simulated anomaly data.