There is a special interaction between the environment, soil microorganisms, and tea plants, which constitute the ecosystem of tea plantations. Influenced by environmental factors and human management, the changes in soil microbial community affected the growth, quality, and yield of tea plants. However, little is known about the composition and structure of soil bacterial and fungal communities in 100-year-old tea plantations and the mechanisms by which they are affected. In this regard, we characterized the microbiome of tea plantation soils by considering the bacterial and fungal communities in 448 soil samples from 101 ancient tea plantations in eight counties of Lincang city, which is one of the tea domestication centers in the world. 16S and Internal Transcribed Spacer (ITS) rRNA high-throughput amplicon sequencing techniques were applied in this study. The results showed that the abundance, diversity, and composition of the bacterial and fungal communities have different sensitivity with varying pH, altitude, and latitude. pH and altitude affect soil microbial communities, and bacterial communities are more sensitive than fungi in terms of abundance and diversity to pH. The highest α-diversity of bacterial communities is shown in the pH 4.50–5.00 and 2,200-m group, and fungi peaked in the pH 5.00–5.50 and 900-m group. Because of environmental and geographical factors, all microbes are similarly changing, and further correlations showed that the composition and structure of bacterial communities are more sensitive than fungal communities, which were affected by latitude and altitude. In conclusion, the interference of anthropogenic activities plays a more important role in governing fungal community selection than environmental or geographical factors, whereas for the bacterial community, it is more selective to environment adaptation than to adaptation to human activities.