Intelligent learning platforms and education information application platforms are gaining ground, owing to the wide application of modern technologies such as the Internet of Things, big data analysis, artificial intelligence, and cloud computing. However, the current platforms cannot solve specific teaching problems, and the relevant research mostly focuses on primary and secondary education. Therefore, this paper constructs and analyzes a framework of intelligent education system for higher education based on the deep learning. Firstly, the functional block diagram of the system was built up. Next, a face detection algorithm was proposed based on the multi-task convolutional neural network, a face recognition algorithm was developed based on the improved deep convolutional neural network, and the knowledge learning status of students was tracked based on the memory augmented neural network. Finally, the proposed framework was proved effective and swift through experiments. The research results expand the application scope of the deep learning in education.