The traditional English teaching mode is that the teacher simply imparts textbook knowledge and students understand and absorb it. However, this method has obvious problems; that is, it is difficult to ensure that students can quickly understand the content taught and cannot get fast feedback. The video feedback method is a method of English teaching combined with audiovisual technology. Teachers can use video technology to record key English knowledge through video and then give feedback and explain the learning through video. At the same time, the intelligent video feedback system teaching method will greatly improve the teaching quality of English classroom and make students’ learning more fun. This paper mainly designs the English teaching system based on the video feedback method and finally realizes the intelligent feedback scheme of the English teaching system. Firstly, the neural network method, classification method, and video shooting technology are used to extract and predict the characteristics of students’ classroom expressions, speech, and so on, and analyze through the video feedback system of English classroom. The research results show that the classification method proposed in this study can better complete the body movements such as student expressions and speech collected by the English teaching system, and the neural network method can more accurately predict and feed back the teaching content through the video feedback method with the largest error. It was only 2.98%, and the linear correlation also reached more than 0.98. The minimum error of the video feedback information is only 0.95%, and the prediction errors of the other two kinds of English classroom information are also 2%. The classification and prediction of intelligent video feedback information have achieved good results.