Spectrum prediction is a promising technique to release spectrum resources and plays an essential role in cognitive radio networks and spectrum situation generating. Traditional algorithms normally focus on one-dimensional or predict spectrum values in a slot-by-slot manner and thus cannot fully perceive the spectrum states in complex environments and lack timeliness. In this paper, a deep learning-based prediction method with a simple structure is developed for temporal–spectral and multi-slot spectrum prediction simultaneously. Specifically, we first analyze and construct spectrum data suitable for the model to simultaneously achieve long-term and multi-dimensional spectrum prediction. Then, a hierarchical spectrum prediction system is developed that takes advantage of the advanced Bi-ConvLSTM and the seq2seq framework. The Bi-ConvLSTM captures time–frequency characteristics of spectrum data, and the seq2seq framework is used for long-term spectrum prediction. Furthermore, the attention mechanism is used to address the limitations of the seq2seq framework that compresses all inputs into fixed-length vectors, resulting in information loss. Finally, the experimental results have shown that the proposed model has a significant advantage over the benchmark schemes. Especially, the proposed spectrum prediction model achieves 6.15%, 0.7749, 1.0978, and 0.9628 in MAPE, MAE, RMSE, and R2, respectively, which is better than all the baseline deep learning models.