In this paper, we use a particle swarm optimization neural network algorithm to analyze the teaching data of physical education faculties and evaluate the quality of teaching in physical education faculties. By studying and analyzing the optimization problem of the weight parameters of convolutional neural network training, this paper designs a hybrid algorithm combining the improved PSO algorithm and the traditional gradient descent in the framework of the BP algorithm by using the gradient information of the loss function and the principle of group cooperative search through PSO algorithm. The hybrid algorithm takes the loss function as the objective function, based on the principle of the PSO algorithm, and optimizes the objective function by combining the gradient information of the loss function of the convolutional neural network. The convergence speed and global search ability of the algorithm are effectively improved while ensuring an acceptable increase in computation. The weight values of the three-level indicators of teacher teaching behavior, student learning behavior, and teaching environment relative to the teaching quality of physical education classroom are 0.106, 0.634, and 0.260, respectively, which shows that the dimension of student learning behavior has the highest weight value in the evaluation of physical education classroom teaching quality, followed by teaching environment and finally teacher teaching behavior. Teachers’ teaching ability will affect the effect of teaching methods, and the stronger the teaching ability is, the better the selection and utilization of teaching methods can be optimized.