With the sustained, rapid, and steady development of China’s economy, the development of higher education is also faster and faster. With economic development and social progress, the demand for talents is also growing. Colleges and universities are the cradle of cultivating excellent talents and undertake the important task of cultivating reserve forces for the country. As an engineer of human soul, English teachers shoulder the responsibility of cultivating excellent talents and the hope of countless parents. For a long time, there have been many problems in the evaluation of College English teachers in China. For example, the assessment objective is too simple, only related to salary promotion and professional title evaluation. In terms of the content of evaluation, they often only pay attention to the quantity and level of scientific research results, ignoring the evaluation of potential, attitude, and other aspects, especially the social responsibility of teachers. Based on the theory of nonlinear random matrix and the theory of competence, this study establishes the quality evaluation model of university teachers in China by combining qualitative and quantitative methods. On this basis, it plays an important role in the selection, training, promotion, and further development of university teachers in China. First of all, according to a large number of domestic and foreign literature on competency, this study summarizes the competency theory and competency model. Using the hypothetical model, a questionnaire was designed and distributed to build a competency model of university teachers from four aspects: professional knowledge, related skills, work attitude, and personality motivation. Finally, the weight of each index in the model is calculated. This study not only constructs the competency evaluation model but also uses the principal component analysis method to calculate it in detail, which provides a good quantitative basis for the selection and performance evaluation of college teachers, so as to prevent the situation beyond its control. It paves an endless bright road for further stable and healthy development. Therefore, the overall research results of this study have greater practical application value and rich connotation.