The traditional mode of listening teaching has been unable to fully adapt to the development trend of the times, and it is also difficult to meet the learning needs of students in the information age. In this paper, the multi-objective optimization algorithm is used to improve the teaching-training process to optimize the problems existing in the teaching of listening course, to provide a multi-objective teaching optimization algorithm to imitate the English listening teaching and learning process and to deal with the non-linear constraints in the multi-objective optimization problem utilizing a penalty function. The multi-objective teaching optimization algorithm is used as the basis to establish a diversified teaching model for college English listening courses, and the effectiveness of the teaching is verified by the listening learning performance, students’ multidimensional ability, and learning emotion. The results show that after 16 weeks of diversified teaching, the mean scores of students’ listening test scores in the experimental group increased by 9.52-24.50 points compared with those before the experiment, the mean value of short-term memory scores was the highest at 27.98, and the Cohen’s d value was above 0.861, and the mean scores of the students’ learning emotion test increased by 1.2938 points compared with those before the implementation. Applying the diversified teaching mode to the teaching of college English listening courses can improve students’ English listening performance, eliminate their anxiety and fear, and promote the diversification of college English listening teaching.