In today’s modern power devices and rapid growth power demands, the need for precise and accurate protection relays is a must for the power distribution system. That is, to segregate faulty sectors within fewer cycles, power relays should perform at the highest level of accuracy to detect abnormal conditions in power distribution. Therefore, this work will investigate the enhancement of the numerical relay testing in terms of harmonic distortions effect on the digitized output waveform as direct causes of relay failures. However, as it is an expensive process of testing the digitizing element of the numerical relay, this paper proposes a new algorithm of Wavelet transforms in power quality signal processing testing using MATLAB simulation. As this newly proposed method of advanced waveform analysis algorithm will enhance the testing process of digitizing elements, and reduce data compiling complexity, a comparison between conventional Fourier Transforms testing and Wavelet algorithm under abnormal conditions will be simulated based on inserting multi harmonics effect. As a result, based on the Wavelet bank of filters, de-noising, and decomposition structure filters, Wavelet has provided promising results in defining the effect of waveform distortion tripping time, fault location, total harmonic distortion, signal-to-noise ratio, and spurious-free dynamic range.