To diagnose damages within the retaining wall structure, the Hilbert marginal energy spectrum was acquired via the Hilbert–Huang transformation of virtual impulse response functions of responses to the retaining wall under ambient excitations. Based on the Hilbert marginal energy spectrum, the Hilbert damage feature vector spectrum was created. On the basis of the damage feature vector spectrum, a damage identification index was proposed. Based on the damage feature vector spectrum and damage index, the damage state of the retaining wall was detected by the damage feature vector spectrum, damage locations of the wall were diagnosed by the damage index trend surface, and the damage intensity of the wall was identified by the quantitative relationship between the damage index and damage intensity. Based on this, a damage diagnosis method for retaining wall structures was proposed. To verify the feasibility and validity of the damage diagnosis method, both model tests and field tests on a pile plate retaining wall are performed under ambient excitations. Test results show that the damage state of the wall can be detected sensitively, damage locations can be diagnosed validly, and damage intensity can be identified quantitatively via this damage diagnosis method.