Aiming at the problem that the traditional model has low accuracy in describing stock excess return, in order to further analyze the change law of stock excess, based on the nonlinear random matrix and esg factor theory, the traditional learning model is analyzed, and the corresponding optimized deep learning model is obtained by introducing the single ring theorem and statistical data. Through the analysis and research of related indexes, the change rules of different indexes are obtained, and the optimization model is used to calculate and forecast the excess return of stocks. The results show that the statistics and spectral radius show typical local linear variation with the increase of eigenvalue. The corresponding statistics show a trend of gradual increase. The corresponding spectral radius has a decreasing variation law, and the two curves have obvious symmetry at some eigenvalues. It can be seen from the change curves under different factors that the change trend of the yield curve is mainly affected by the investment factor, while the change rule of the specific value of the yield curve is controlled by the profit factor. This shows that the two factors have the same influence on the stock excess return. The influence of optimized deep learning model on stock excess index has typical linear characteristics, which can be divided into linear increase and linear decline according to different change rules. The basic type has the greatest influence, while the corresponding pattern analysis type has the least influence. Finally, the method of experimental verification is used to verify the stock excess data, and the results show that the optimized deep learning model can better characterize the experimental results. Therefore, the optimized deep learning model based on nonlinear random matrix and esg factor can carry out targeted analysis of different types of stock returns, thus improving research ideas and calculation methods for the application of deep learning model in different fields.