Educational data mining is becoming a more and more popular research field in recent years, mainly with the help of cross research conducted by various disciplines, so as to solve various difficult problems in the teaching and education process. In this paper, we proposed a hybrid approach for student performance prediction. We collected the dataset, including 15 characteristics of students from three categories (individual basic information, individual education information, and individual behavior information). Based on the random forest (RF) and simulated annealing (SA) algorithms, we binary encode the relevant parameters (number of features, tree size, and tree decision weights) as the target variables for algorithm optimization, use the out-of-bag error as the optimization objective function, and then propose the IRFC (improved random forest classifier) algorithm in this paper. Compared with other mainstream improved random forest algorithms, the research results demonstrate that the proposed algorithm in this paper has higher generalization ability and smaller OOB error. This study provides a methodological reference for the prediction of student achievement and also makes a marginal contribution to student management work.