It is necessary to assess damage properly for the safe use of a structure and for the development of an appropriate maintenance strategy. Although many efforts have been made to measure the vibration of a structure to determine the degree of damage, the accuracy of evaluation is not high enough, so it is difficult to say that a damage evaluation based on vibrations in a structure has not been put to practical use. In this study, we propose a method to evaluate damage by measuring the acceleration of a structure at multiple points and interpreting the results with a Random Forest, which is a kind of supervised machine learning. The proposed method uses the maximum response acceleration, standard deviation, logarithmic decay rate, and natural frequency to improve the accuracy of damage assessment. We propose a three-step Random Forest method to evaluate various damage types based on the results of these many measurements. Then, the accuracy of the proposed method is verified based on the results of a cross-validation and a vibration test of an actual damaged specimen.