Soil microbial diversity is important for maintaining ecosystem functions. However, the linkage between microbial diversity, especially rare and abundant bacterial diversity, and carbon decomposition remains largely unknown. In this study, we assessed the establishment and maintenance of rare and abundant bacterial α-diversities at the taxonomic and phylogenetic levels and their linkages with soil carbon decomposition separately in four Chinese woodlands. Compared to abundant bacteria, rare bacteria showed higher community diversity, tighter phylogenetic clustering, wider environmental breadth, stronger phylogenetic signals, and higher functional redundancy. The assembly of the abundant bacterial subcommunity was governed by stochastic (59.2%) and deterministic (41.8%) processes, whereas the assembly of the rare bacterial subcommunity was mainly dominated by deterministic processes (85.8%). Furthermore, total phosphorus, soil pH, and ammonium nitrogen balanced stochastic and deterministic processes in both rare and abundant bacterial subcommunities. Our results reveal that rare bacteria displayed stronger environmental adaptability and environmental constraint. Importantly, the α-diversities of rare taxa, rather than abundant taxa, were significantly related to carbon decomposition. This study provides a holistic understanding of biogeographic patterns of abundant and rare bacteria and their α-diversities in relation to carbon decomposition, thus helping us better predict and regulate carbon dynamics under the background of global climate change.