Under long-term load, the creep deformation of concrete materials has a serious impact on the structural safety of hydraulic structures, especially under the action of ultra-high stress levels, the concrete materials will undergo nonlinear creep, which is extremely easy to cause structural damage. In this study, the uniaxial nonlinear creep test of concrete specimens was used to establish the damage index based on the wave velocity value of ultrasonic flaw detection, and the creep and damage degree curve of the concrete specimen were obtained. The ideal elastic element, the Kelvin body, and the nonlinear viscoplastic element are connected in series, and a new viscoelastic–plastic model considering the creep characteristics of concrete is proposed. Based on the principle of least squares, the Levenberg–Marquardt (LM) algorithm is used to inverse the parameters of the nonlinear creep test. In addition, the model is verified by the measured data of linear creep. At the same time, the sensitivity of each model parameter is analyzed. The research shows that the LM algorithm can give the fitting parameters of the model better and faster, and the fitting values of the model are similar to the experimental results. The sensitivity analysis of the parameters shows that the proposed model has good stability and good adaptability. The model has a more accurate description of the various stages of creep, and may be conveniently applied to concrete creep calculations in actual projects.