A Gauss process state-space model trained in a laboratory cannot accurately simulate a nonlinear system in a non-laboratory environment. To solve this problem, a novel Gauss process state-space model optimization algorithm is proposed by combining the expectation–maximization algorithm with the Gauss process Rauch–Tung–Striebel smoother algorithm, that is, the EM-GP-RTSS algorithm. First, a theoretical formulation of the Gauss process state-space model is proposed, which is not found in previous references. Second, a Gauss process state-space model optimization framework with the expectation–maximization algorithm is proposed. In the expectation–maximization algorithm, the unknown system state is considered as the lost data, and the maximization of measurement likelihood function is transformed into that of a conditional expectation function. Then, the Gauss process–assumed density filter algorithm and the Gauss process Rauch–Tung–Striebel smoother algorithm are proposed with the Gauss process state-space model defined in this article, in order to calculate the smoothed distribution in the conditional expectation function. Finally, the Monte Carlo numerical integral method is used to obtain the approximate expression of the conditional expectation function. The simulation results demonstrate that the Gauss process state-space model optimized by the EM-GP-RTSS can simulate the system in the non-laboratory environment better than the Gauss process state-space model trained in the laboratory, and can reach or exceed the estimation accuracy of the traditional state-space model.