Fermentation is a key link in determining the quality and flavor formation of black tea. However, during the actual production, the judgment of black tea fermentation quality mainly relies on the sensory evaluation of the tea maker, which is more subjective and prone to cause inconsistency in tea quality. Traditional testing methods, such as physical and chemical analyses, are time-consuming, laborious, and costly and are unable to meet the needs of the actual production. In this study, a self-developed machine vision system was used to quickly and accurately identify the degree of black tea fermentation by acquiring color and texture information on the surface of fermented leaves. To accurately control the quality of black tea fermentation and to understand the dynamic changes in key endoplasmic components in the fermented leaves, a quantitative prediction model of the key endoplasmic components in the fermentation process of black tea was constructed. The experiments proved that the system achieved 100% accuracy in discriminating the degree of fermentation of black tea, and the prediction accuracy of catechin components and thearubigin content reached more than 0.895. This system overcomes the defects of accurate measurement of multiple sensors coupled together, reduces the detection cost, and optimizes the experimental process. It can meet the needs of online monitoring in actual production.