Displacement signals, acquired by eddy current sensors, are extensively used in condition monitoring and health prognosis of electromechanical equipment. Owing to its sensitivity to low frequency components, the displacement signal often contains sinusoidal waves of high amplitudes. If the digitization of the sinusoidal wave does not satisfy the condition of full period sampling, an effect of severe end distortion (SED), in the form of impulsive features, is likely to occur because of boundary extensions in discrete wavelet decompositions. The SED effect will complicate the extraction of weak fault features if it is left untreated. In this paper, we investigate the mechanism of the SED effect using theories based on Fourier analysis and wavelet analysis. To enhance feature extraction performance from displacement signals in the presence of strong sinusoidal waves, a novel method, based on the Fourier basis and a compound wavelet dictionary, is proposed. In the procedure, ratio-based spectrum correction methods, using the rectangle window as well as the Hanning window, are employed to obtain an optimized reduction of strong sinusoidal waves. The residual signal is further decomposed by the compound wavelet dictionary which consists of dyadic wavelet packets and implicit wavelet packets. It was verified through numerical simulations that the reconstructed signal in each wavelet subspace can avoid severe end distortions. The proposed method was applied to case studies of an experimental test with rub impact fault and an engineering test with blade crack fault. The analysis results demonstrate the proposed method can effectively suppress the SED effect in displacement signal analysis, and therefore enhance the performance of wavelet analysis in extracting weak fault features.