Learner’s Emotional state has an important impact on affective and cognitive processes for classroom teaching. It is quite necessary to detect learner’s emotion state unconsciously in teaching and learning processes. A fast facial expression recognition algorithm is presented to detect the emotional state of the learner in real learning environment. Gabor convolutional network (GCN) is used to classify the facial expression. The image extracted from teaching and learning environment need to be preprocessed for accelerating the expression recognition. A skin color segmentation model, generalized Gaussian mixture distribution (GGMD), is designed by using expectation and maximization (EM) algorithm to detect the facial area rapidly. Then a fast facial expression recognition algorithm is designed by using the skin color model and the GCN. Experiment results show the satisfied accuracy and excellent time performance of the system.