Sports psychology is a branch of psychology and is a sports course that takes both theory and practice into account. It is not only an integral part of the talent training curriculum system of sports colleges and universities, but also a basic compulsory course for social sports majors in comprehensive universities. Sports psychology has currently become one of the significant theories in PE teaching, which not only guides PE teaching, but also helps solve existing problems in PE teaching and drives the development of PE teaching. Subject to the impact of traditional education mode and educational concept, etc., however, there are some problems in the existing the teaching of sports psychology, such as students’ low interest, poor initiative and poor educational effect. To raise students’ learning interest, this paper selected a problem-oriented modular teaching mode, and applied it to the sports psychology classroom, to improve the learning effect. In this study, we first used data mining technology to build an anxiety prediction model and then identified a problem to be solved. After that, the problem was designed and analyzed, the known problem was confirmed, new information was collected, old and new knowledge was integrated, until the problem was finally solved. On this basis, based on virtual simulation technology, an experimental simulation teaching system for sports psychology was presented. It was found through research that the problems-oriented teaching mode can significantly enhance the effect of sports psychology and PE teaching and increase students’ learning initiative. The learning effect was significantly better than that of traditional teaching methods.