Residual stress is closely related to the evolution process of the component fatigue state, but it can be affected by various sources. Conventional fatigue evaluation either focuses on the physical process, which is limited by the complexity of the physical process and the environment, or on monitored data to form a data-driven model, which lacks a relation to the degenerate process and is more sensitive to the quality of the data. This paper proposes a fusion-driven fatigue evaluation model based on the Gaussian process state–space model, which considers the importance of physical processes and the residuals. Through state–space theory, the probabilistic space evaluation results of the Gaussian process and linear physical model are used as the hidden state evaluation results and hidden state change observation function, respectively, to construct a complete Gaussian process state–space framework. Then, through the solution of a particle filter, the importance of the residual is inferred and the fatigue evaluation model is established. Fatigue tests on titanium alloy components were conducted to verify the effectiveness of the fatigue evaluation model. The results indicated that the proposed models could correct evaluation results that were far away from the input data and improve the stability of the prediction.