Establishing a perfect teaching evaluation system is one of the essential links to test the teaching results. In this paper, firstly, on the basis of clarifying the principle of constructing the evaluation index system of spoken English teaching, we formulate the index elements of the first, second level, and third levels, build the evaluation index system of spoken English teaching through the Delphi method, and initially determine the degree of importance of the evaluation indexes by relying on the Likert scale. Then the indicators are assigned with the compatibility matrix analysis method and the improved entropy weighting method, and the level of spoken English teaching in colleges and universities is quantitatively analyzed from the two primary levels of students and teachers through the fuzzy comprehensive evaluation method. Finally, examples are used to examine students’ readiness to evaluate spoken English teaching in colleges and universities, and the effectiveness of the constructed evaluation system for spoken English teaching is assessed. The results show that the total score of students’ rating of spoken English teaching in colleges and universities is 86.5, and the total score of teachers’ rating is 93.3, both of which are more than 85, indicating that the quality of spoken English teaching in the sample colleges and universities is good. This study realizes further improvement and transformation at the levels of evaluation concept, evaluation content, evaluation method, etc.. It provides specific references for teaching evaluation and decision-making regarding teaching reform in other disciplines.