PurposeNumerous educational applications (APP) have been developed to assist traditional classroom teaching and student learning. APP quality plays a critical role in influencing students' learning behaviors. However, the role negative mindsets, especially computer anxiety, play in how APP quality affects student engagement remains unknown. To address the relationships among APP quality, computer anxiety and student engagement in an APP-based learning environment, we developed an extended information system (IS) success model that includes interface and instructor quality.Design/methodology/approachTo empirically test the proposed model, we conducted a survey with a sample of 225 university students and examined the hypotheses using the partial least squares (PLS) method.FindingsComputer anxiety was demonstrated to fully mediate the relationships between student engagement and interface quality and service quality and system quality. In addition, the instructor quality acts as a partial mediator of the relationship between computer anxiety and student engagement.Originality/valueThis study reveals the important mediating role of computer anxiety in APP-assisted learning and the special status of instructor quality and user experience in influencing student engagement. The findings of this study shed meaningful light on the practical implications for instructors and APP software developers.