Effective teaching behavior in physical education has an important impact on the quality of classroom teaching. To overcome the shortcomings in the existing university teaching quality assessment procedure, this paper designs an evaluation model for physical education departments based on long short-term memory networks (LSTM) with the improved particle swarm optimization (PSO). The model is constructed by analyzing the connotation and index system of teacher education teaching quality and constructing a fan-leaf structure model of teacher education teaching quality. Then, an improved PSO-LSTM model is proposed to train the teaching samples. The evaluation model applies the improved LSTM model and optimizes the network structure by dynamically adjusting the learning rate. The model is then optimized for the number of neurons and iterations of the network using the improved PSO. The results of the experiment indicate that the proposed model is effective in evaluating the quality of physical education. Moreover, the model analysis’ accuracy has greatly improved. This helps teachers have a comprehensive understanding of classroom dynamics and improve their professional competence and classroom teaching quality.