Modal analysis constitutes a fundamental aspect of structural investigation within diverse engineering domains, encompassing sectors such as automotive, wind energy, and aerospace. The prominence of high-frequency excitation loads, exemplified by the combustion phenomena in liquid rocket engines, necessitates an in-depth examination of the high-frequency vibrational response within structural components. However, the complexity of evaluating high-frequency vibrations arises from the negligible displacement associated with these responses. When using an optical full-field measurement system based on a high-speed camera for vibration measurement, it is usually severely affected by noise. Direct analysis of raw data using an optical measurement system (3D-DIC) is not feasible. In this paper, we combine phase-based motion magnification and digital image correlation methods to obtain the high-frequency vibration modes of the structure. 3D-DIC(3D Digital Image Correlation)analysis is performed on the magnified images to quantify the out-of-plane vibration modes of the structure. Using the cantilever beam as an example, the first five out-of-plane vibration mode shapes were separated from the response video under a single hammer excitation. Especially the 5th order natural frequency is as high as 3503Hz, and the corresponding structural response was below the noise floor of the camera system. The vibration mode results obtained by this method are highly consistent with the vibration modes obtained by the 3D-SLDV(3D Scanning Laser Doppler Vibrometer) method. Finally, this method was applied to identify the out-of-plane vibration modes of real engine pipe. The combination of motion magnification techniques and DIC can enhance the capability of traditional 3D-DIC, which is beneficial for high-frequency structural identification. Future research could concentrate on optimizing motion amplification factors for different structures and loads, and creating automated algorithms for analyzing and visualizing amplified motion data in real time.