Deep learning can remove the noise of the terahertz (THz) spectrum via its powerful feature extraction ability. However, this technology suffers from several limitations, including clean training data being difficult to obtain, the amount of training data being small, and the restored effect being unsatisfactory. In this paper, a novel THz spectrum denoising method is proposed. Low-quality underwater images and transfer learning are used to alleviate the limitation of the training data amount. Then, the principle of Noise2Noise is applied to further reduce the limitations of clean training data. Moreover, a THz denoising network based on Transformer is proposed, and fractional variation is introduced in the loss function to improve the denoising effect. Experimental results demonstrate that the proposed method estimates the high-quality THz spectrum in simulation and measured data experiments, and it also has a satisfactory result in THz imaging.