In this paper, we propose to mine the class videos and analyze the behaviors of students to obtain the information on the focus of the students during one class. As a methodological contribution, we investigate face detection, tracking, and verification techniques to measure student engagement from the video records. We also analyze several behaviors of a single student or groups of students, such as temporal behaviors and spatial behaviors. Furthermore, we mine the relationship between students' behaviors and grades throughout the whole semester, including 16 weeks. This work can help teachers to improve the quality of class, find students who tend to get bad grades and give these students timely help. Experimental results on the test videos of different kinds of classes demonstrate the effectiveness of the proposed method.