In the era of big data, analyzing students’ classroom behavioral status is an effective means of enhancing teaching quality and performance. This paper proposes algorithmic research on student behavior recognition and detection, researches the related algorithms of convolutional neural network’s target detection, further improves the student behavior recognition algorithm of YOLOv5, and tests the optimization program of lacquer art teaching as a real-world test research in terms of students’ behaviors and teaching effects. After using the behavior recognition algorithm proposed in this paper, the students achieved certain results in the creative learning of lacquer artworks. The student’s final grades in lacquer art have a good improvement; the mean value of the final grades of the experimental class is 89.43, the mean value of the final grades of the control class is 85.32, and the mean value of the final grades of the experimental class is higher than that of the control class by 4.11. It can be concluded that the use of learning behavioral recognition algorithms in the teaching of lacquer art has a good effect on enhancement.