This study introduces a novel Unmanned Aerial Vehicle-mounted (UAV-mounted) Laser Doppler Vibrometer (LDV) system for detecting debonding damage in Hidden Frame-Supported Glass Curtain Walls (HFSGCW). The established system enables UAVs to transport the LDV to high altitudes for operation. The vibration signals acquired by the UAV-LDV system are decomposed into different energy bands by wavelet packet analysis, and then the occurrence and location of the damage are identified by the Sum of Squared Differences (SSD) of the wavelet packet bands’ energy. This paper investigates the potential factors affecting the performance of the Unmanned Aerial Vehicle-Laser Doppler Vibrometer (UAV-LDV) system, including the arrangement of measuring points, measuring distance, noise level, and wind speed through the first-order natural frequency, the normalized frequency response functions, and the SSD indicator. Experimental and simulation results confirm the effectiveness of the UAV-LDV system, highlighting its advantages over traditional methods by offering remote, non-contact, and efficient debonding detection. This method not only indicates the presence of the damage, as traditional indicators do, but also pinpoints the exact location of it, ensuring safety and cost-effectiveness in high-rise inspections. The proposed method and indicator offer advantages in terms of convenience, visualization, and efficiency. The study discusses the impact of measurement point arrangement, measuring distance, noise levels, and wind speed on the system’s performance. The findings demonstrate that while the UAV-LDV system introduces new capabilities in rapid and reliable structural damage assessment, operational challenges such as wind and noise levels significantly influence its accuracy.