In the context of digital intelligence empowerment, the digital literacy level of tea farmers has a significant impact on the intelligent development and transformation of the tea industry. This study extends the original model of the unified theory of acceptance and use of technology (UTAUT) by introducing the personal innovativeness theory and the self-efficacy theory and constructs a new model to explore the influencing factors of moderately scaled tea farmers’ digital literacy improvement behavior. There are a total of 22 research hypotheses. Using structural equation modeling and collecting questionnaire data for analysis, the following research results were obtained. (1) The performance expectancy, social influence, effort expectancy, personal innovativeness, and self-efficacy all significantly positively affected the willingness of tea farmers to improve their digital literacy, according to the path coefficient in descending order: social influence (0.226) > self-efficacy (0.224) > effort expectancy (0.178) > performance expectancy (0.157) > personal innovativeness (0.155). (2) Facilitating conditions and the willingness to improve digital literacy had a significant positive impact on tea farmers’ digital literacy improvement behavior, according to the size of the path coefficient: the willingness to improve (0.271) > facilitating conditions (0.106). (3) The willingness of tea farmers to improve their digital literacy played a complete mediating role between personal innovativeness and self-efficacy on their digital literacy improvement behavior, and was partially mediated between the performance expectancy, social influence, and effort expectancy on their digital literacy improvement behavior. According to the proportion of indirect effects, the order was effort expectancy (27%), performance expectancy (47%), and social influence (49%). (4) The gender and age of tea farmers had a significant positive moderating effect on the impact of performance expectancy on the willingness to improve digital literacy. Age and experience had a significant positive moderating effect on the impact of effort expectancy on the willingness to improve digital literacy. The age of tea farmers had a significant positive moderating effect on the improvement of digital literacy behavior through the facilitating conditions. This study extended the applicability of the UTAUT theoretical model and proposed six strategies to improve the digital literacy of tea farmers, which helps policymakers and industry leaders provide practical guidance for tea farmers to improve their digital literacy and provide reference for research related to farmers’ digital literacy.