Soil microorganisms found in shrub-meadow ecosystems are highly heterogeneous and extremely sensitive to grazing, but changes in microbial compositional and functional heterogeneity during grazing exclusion (GE) have been largely overlooked compared to community diversity. We collected soil samples from heavily grazed plots (6.0 sheep/ha) and GE plots (matrix and patch areas in both), and used a combination of next-generation sequencing, vegetation features, and the associated soil property data to investigate the effect of GE on the composition and function of microbial communities (bacteria fungi, and archaea) in 0–10 cm soils. Regarding community composition, the proportions of species in bacteria, fungi, and archaea were 97.3, 2.3, and 0.4%, respectively. GE significantly affected the species diversity of fungi and archaea but not that of bacteria. GE decreased the heterogeneity of bacteria (2.9% in matrix and 6.2% in patch) and archaea (31.1% in matrix and 19.7% in patch) but increased that of fungi by 1.4% in patch. Regarding community function, enzyme diversity and heterogeneity were increased by 10.4 and 9.4%, respectively, in patch after 6 years of fencing, exemplifying a high level of microbial functional redundancy. The Kyoto Encyclopedia of Genes and Genome pathways—cell growth and death, translation, digestive system, and nucleotide metabolism—were functional biomarkers (linear discriminant analysis effect size method) in matrix-non-grazed plots, whereas lipid metabolism, xenobiotics biodegradation and metabolism, and metabolism of terpenoids and polyketides, cell motility, cancer: overview, endocrine system, and membrane transport were biomarkers in patch-non-grazed plots. Additionally, GE improved the capacity for fatty acid metabolism but decreased the abundance of methane-producing archaea by 42.9%. Redundancy analysis revealed that the factors that affected microbial composition the most were soil aggregates, soil moisture, and the number of plant species, whereas those that affected microbial function the most were soil available phosphorus, soil temperature, and shrub canopy diameter. Our results quantified soil microbial heterogeneity, emphasizing the different responses of the composition and function of bacteria, fungi, and archaea to GE in alpine shrubs and meadows.