The impact-echo diagnostic method is a well-known nondestructive pulse compression test method, which can be relatively easily used for the testing of concrete and reinforced concrete elements. The evaluation of the measurement with this method is based on the analysis of the signal itself in the time and frequency domains. This allows acquisition of information on the velocity of the mechanical wave, the resonant frequency of the specimen or on the presence of internal defects. The ability to interpret these measurements depends on the experience of the diagnostic technician. The advent of classification algorithms in the field of machine learning has brought an increasing number of applications where the entire interpretation phase can be considerably simplified with the help of classification models. However, this automated evaluation procedure must be provided with the information of whether the signal acquired by the test equipment has actually been measured under optimally set conditions. This paper proposes a procedure for the mutual comparison of different measuring setups with a variable tip type, hammer handle and impact force. These three variables were used for a series of measurements which were subsequently compared with each other using multi-criteria evaluation. This offers a tool for the evaluation of measured data and their filtering. As an output of the designed method, each measurement is marked by a score value, which represents how well the acquired signal fit the weight demands for each observed feature of the signal. The method allows the adjustment of selected demands for a specific application by means of set thresholds. This approach enables the understanding of characteristics of the signal in the automated pre-processing of measured data, where computing power is limited. Thus, this solution is potentially suitable for remote long-term observations with sensor arrays or for acoustic emission signals pre-processing.