By improving the logistic regression function and selecting a step-by-step fitting result using the least square method as the input of the logistic regression model, this paper analyzes the situation of students failing the course. Compared with arctan-exp function and sigmoid function, the former has better robustness and stability and makes the results tend to 0 and 1 and be classified. An improved algorithm based on arctan − exp logistic regression and least square method, which combines the advantages of both functions, is studied. Finally, an implementation algorithm is presented to well meet the function. Besides, through a simulation example, both theoretical analysis and experimental evaluation demonstrate the effectiveness of our proposed approach, and it shows that the nonlinear arctan-exp function, which bases on the least square method, is used as the distribution function to predict the effectiveness of students’ failure. The algorithm has been compared and evaluated, which obtains superior results in terms of both accuracy rate and recall rate of the diagnosis results of the students failing.