Background Effective prevention and treatment of cow mastitis can provide a good guarantee for the healthy growth of cows and the qualified production of dairy products. The main purpose of this study was to explore the effect of tea tree oil on lipopolysaccharide (LPS) -induced mastitis in dairy cows, and the key gene in LPS -stimulated bovine mammary epithelial cells (BMECs) was identified. Results In this study, a model of mastitis induced by LPS was constructed, to which tea tree oil and LPS were added. The protective effects of tea tree oil on LPS-induced mastitis in BMECs were verified by the CCK-8 method, flow cytometry, real-time fluorescence quantitative detection, ELISA and other methods. The results showed that LPS at a concentration of 200 μg/ml could reduce the proliferative activity of the cells, induce a high proportion of apoptosis, and promote the expression of TNF-α, IL-6 and STAT1. Upon addition of tea tree oil, the proportion of apoptosis was reduced, and the expression of NF-κB, MAPK and caspase-3 was inhibited. Mammary epithelial cells were compared under control and LPS-treatment conditions and analyzed by second-generation sequencing. A total of 1270 mRNAs were identified as differentially expressed, of which 787 genes were upregulated and 483 were downregulated. These differentially expressed genes include TNF - α, IL6, STAT1, mapk4, etc. H&E staining and immunohistochemistry were used to verify the function of candidate genes. TNF-α and IL6 were observed to play important roles in mediating the preventive effect of tea tree oil on mastitis in LPS-stimulated bovine mammary epithelial cells. Conclusions The results showed that tea tree oil had a protective effect against LPS-induced mastitis. TNF - α and IL6 may be the marker genes of LPS-induced mastitis which provided a theoretical basis and experimental support for further research to determine new strategies for the prevention and treatment of mastitis and improvement of milk quality.