Residual stresses, mainly generated by mechanical processing and strengthening approaches, are known to impose significant influence on the performance of industrial structures. They usually exist as an undesirable source of fractures and failures. Furthermore, the nondestructive evaluation and calibration of residual stresses in the depth direction always stays a significant issue, especially for multilayer viscoelastic composites. Thus, this paper proposes the advancement of residual stress evaluation technology by detecting the variation of guided wave signal and impedance features under different stress conditions. The target structure for nondestructive evaluation task is a typical multilayer, viscoelastic composite shell of a solid rocket motor. By carrying out an in-depth theoretical analysis with Finite Element Modeling (FEM), the sensing sensitivity of the probing ultrasonic wave characteristics are studied systematically. To monitor the residual stress state, a theoretical model is established by considering the stress gradient in each layer, indicating the current stress condition inside the testing object. Consequently, via the comprehensive consideration of transducer dimensions, center frequency, the most effective residual stress monitoring setup is established and the sensing signals are recorded by piezoelectric wafer active sensors (PWAS) placing on the object. Finally, different values of prestress are applied to the object, while the nonlinear constitutive law of the layered materials is implemented. The pattern between the sensing signal and the residual stress state is identified. Simulation results show that significant nonlinear features such as higher harmonic generation are fully captured, which are related to the prestress conditions. The amplitude of sensing signals could provide indicative information for evaluating the residual stress state based on the acoustoelastic effects. The amplitude and the peak shift of the impedance spectra are investigated to quantify the residual stress state. The findings of this research possess superb application potential for the evaluation of residual stresses in multilayer composites for enhancing manufacturing quality and avoiding unexpected failures in solid rocket motor industry. This paper finishes with summary, concluding remarks, and suggestions for future work.