To simulate the condition of unstable rock formation in the stable rock formation of the roadway roof, the complete and broken rock formations on the roadway roof were simulated by superimposing and combining complete specimens and broken specimens, hence providing an appropriate way to obtain the drilling vibration signals to find regular patterns which can be used to judge the state of the rock layers. The Fourier transform analysis method and the wavelet transform analysis method are, respectively, used for the vibration signals to extract the eigenvalues of the vibration signals that are quite different in the complete layer and broken layers. Effective values, kurtosis factor, pulse factors, and gravity amplitude were used to construct the eigenvectors, which were most suitable as the basis for judging the integrity of rock formation after analysis, providing an effective way to deal with drilling signal. The wavelet threshold denoising could adaptively reduce the noise of the signals, and the unbiased likelihood estimation threshold rule had the best denoising effect. Well, through the extraction and comparison of the wavelet feature-scale entropy values, it is concluded that the wavelet analysis can be used as a reference rather than a decisive factor to determine the complete and broken layers. And more research is needed to determine the selection method of wavelet.