Understanding the effects of environmental factors on microbial communities is critical for microbial ecology, but it remains challenging. In this study, we examined the diversity (alpha diversity) and community compositions (beta diversity) of prokaryotes and fungi in hypersaline sediments and salinized soils from northern China. Environmental variables were highly correlated, but they differed significantly between the sediments and saline soils. The compositions of prokaryotic and fungal communities in the hypersaline sediments were different from those in adjacent saline–alkaline soils, indicating a habitat-specific microbial distribution pattern. The macroelements (S, P, K, Mg, and Fe) and Ca were, respectively, correlated closely with the alpha diversity of prokaryotes and fungi, while the macronutrients (e.g., Na, S, P, and Ca) were correlated with the prokaryotic and fungal beta-diversity (P ≤ 0.05). And, the nine microelements (e.g., Al, Ba, Co, Hg, and Mn) and micronutrients (Ba, Cd, and Sr) individually shaped the alpha diversity of prokaryotes and fungi, while the six microelements (e.g., As, Ba, Cr, and Ge) and only the trace elements (Cr and Cu), respectively, influenced the beta diversity of prokaryotes and fungi (P < 0.05). Variation-partitioning analysis (VPA) showed that environmental variables jointly explained 55.49% and 32.27% of the total variation for the prokaryotic and fungal communities, respectively. Together, our findings demonstrate that the diversity and community composition of the prokaryotes and fungi were driven by different macro and microelements in saline habitats, and that geochemical elements could more widely regulate the diversity and community composition of prokaryotes than these of fungi.