Fuzzy English text recognition will be affected by the complex background, resulting in low recognition accuracy. In order to improve the effect of English text recognition, it is necessary to remove the background. Based on the machine vision algorithm, this paper improves the traditional genetic algorithm to obtain an adaptive genetic algorithm, and builds an English text background elimination model based on the adaptive genetic algorithm. Moreover, this paper introduces the hill-climbing method with better local optimization effect to perform local optimization in the iterative process of genetic algorithm, and constructs an adaptive genetic algorithm based on the hill-climbing method to make up for the defects of local optimization in the evolution process of genetic algorithm. In addition, this paper constructs a projection pursuit clustering model based on hill-climbing adaptive genetic algorithm, and constructs the functional module of the English text background elimination model based on actual needs. Finally, this paper designs a control experiment to verify the performance of the model. The research results show that the effect of the model constructed in this paper is good.