Structural health monitoring of grouted sleeves is one of the assembly industry’s huge challenges. In this study, a combined two-level damage detection was introduced. It comprises defect classification (healthy, rebar eccentricity, and grout leakage) and severity evaluation for early-age grouted sleeves using guided waves. Multiple features (MF) from time-, frequency-, and time-frequency domains were extracted and defined according to the diverse defects and ages of grouted sleeves to represent complex damage characteristics. Moreover, the egret swarm algorithm optimization–extreme learning machine (ESAO-ELM) models were proposed to avoid the influence of subjective experience and judgment from experts. ESAO optimized the initial random parameters (input weights and hidden layer bias) of ELM. Then, two MF-ESAO-ELM models were trained for two-level damage detection on the experimental dataset. The performance of the proposed models was comprehensively evaluated using accuracy, recall, precision, and confusion matrix. MF-ESAO-ELM performs better than ELM and PSO-ELM in accuracy. In this strategy, the defect classification model works in the outer layer to distinguish the state and types of defects of grouted sleeves (healthy, eccentric, or leakage). In comparison, the inner layer starts predicting the severity only if the defect type is leakage. MF-ESAO-ELM offers advantages in terms of accuracy, strategy, and calculation time.