With increasing requirements for structural stability and durability, effective monitoring strategies for existing and potential damage are necessary. A laser Doppler vibrometer on moving platforms (LDVom) can remotely capture large-scale structural vibrations, but speckle noise, a significant signal issue mainly when {one-way continuously} scanning from moving platforms, restricts its applications. A novel approach based on ensemble empirical mode decomposition (EEMD) is proposed to eliminate speckle noise. Moving root-mean-square (MRMS) thresholds are used to cut off signal drop-outs. With both numerically simulated and experimentally acquired signals, the proposed EEMD-based approach reveals the true vibrations despite the low initial signal-to-noise ratio (SNR). Other methods fail to eliminate the speckle noise. In physical experiments, the despeckled signal energy is concentrated at defect locations in the Hilbert-Huang spectrum. The identified damage locations agree well with the actual damage locations. Therefore, the developed approach demonstrates advantages and robustness of eliminating speckle noise in LDVom signals for damage inspection.