In recent years, many studies have been conducted on the vision-based displacement measurement system using an unmanned aerial vehicle, which has been used in actual structure measurements. In this study, the dynamic measurement reliability of a vision-based displacement measurement system using an unmanned aerial vehicle was examined by measuring various vibrations with a frequency of 0 to 3 Hz and a displacement of 0 to 100 mm. Furthermore, free vibration was applied to model structures with one and two stories, and the response was measured to examine the accuracy of identifying structural dynamic characteristics. The vibration measurement results demonstrated that the vision-based displacement measurement system using an unmanned aerial vehicle has an average root mean square percentage error of 0.662% compared with the laser distance sensor in all experiments. However, the errors were relatively large in the displacement measurement of 10 mm or less regardless of the frequency. In the structure measurements, all sensors demonstrated the same mode frequency based on the accelerometer, and the damping ratios were extremely similar, except for the laser distance sensor measurement value of the two-story structure. Mode shape estimation was obtained and compared using the modal assurance criterion value compared with the accelerometer, and the values for the vision-based displacement measurement system using an unmanned aerial vehicle were close to 1. According to these results, the vision-based displacement measurement system using an unmanned aerial vehicle demonstrated results similar to those of conventional displacement sensors and can thus replace conventional displacement sensors.