PurposeThis study has two objectives: to explore the factors that influence student self-efficacy regarding engagement and learning outcomes in a business simulation game course and to compare the difference between hierarchical and general teaching methods.Design/methodology/approachFrom September 2021 to May 2022, a questionnaire was administered to 126 students in a business simulation game course at the Zhongshan Institute, University of Electronic Science and Technology of China. Data were analyzed using nonparametric paired samples tests and linear regression.FindingsThe results showed that student self-efficacy, engagement and learning outcomes were significantly higher with the hierarchical teaching method than with the general teaching method. There were also differences in the factors that influenced self-efficacy regarding learning outcomes between the two teaching methods. With the general teaching method, student self-efficacy did not directly affect learning outcomes, but did so indirectly by mediating the effect of engagement. However, with the hierarchical teaching method, self-efficacy directly and significantly affected learning outcomes, in addition to indirectly affecting learning outcomes through student engagement.Research limitations/implicationsCompared with the control group experimental research method, the quasi-experimental research method can eliminate the influence of sample heterogeneity itself, but the state of the same sample may change at different times, which is not necessarily caused by the hierarchical teaching design.Practical implicationsBased on the results of this study, teachers can apply hierarchical teaching according to student ability levels when integrating business simulation games. The results of this study can inspire teachers to protect student self-confidence and make teaching objectives and specific requirements clear in the beginning of the course, and also provide an important practical suggestion for students on how to improve their course performance.Social implicationsThe research results can be extended to other courses. Teachers can improve students' self-efficacy through hierarchical teaching design, thus improving students' learning performance and also provide reference value for students to improve their learning performance.Originality/valueThis study built a model based on self-system model of motivational development (SSMMD) theory, comparing factors that affect student self-efficacy regarding learning outcomes under different teaching methods. The model enriches the literature on SSMMD theory as applied to business simulation game courses and adds to our understanding of hierarchical teaching methods in this field. The results provide a valuable reference for teachers that can improve teaching methods and learning outcomes.