Rock fragments (RFs) are abundant soil constituents, but are routinely excluded from soil analyses. Hence, their contribution to soil properties, and in particular to the microbiome, is incompletely understood. Therefore, shifts in microbial colonisation along the rock‐to‐soil continuum of topsoils from three agricultural sites with different sedimentary parent rock materials were investigated with particular attention to RFs. Microbial biomass and community composition were quantified using phospholipid fatty acid (PLFA) analysis for unweathered and weathered parent rock materials, two RF fractions (8–16 mm and 2–8 mm) and the fine earth (FE; <2 mm). Trends in biogeochemical weathering, nutrient availability and soil organic matter (OM) development were assessed using mineralogical, geochemical and physical analyses. Actinobacterial PLFA was particularly abundant in parent rocks, where Actinobacteria likely contribute to rock weathering and the initiation of OM accumulation. Conversely, bacterial PLFAs were most abundant in the FE under nutrient‐ and OM‐rich conditions. The integral role of RFs as a microbial habitat is demonstrated by a distinct fungal colonisation, which is enabled by the specific physical features of RFs in combination with the provision of inorganic nutrients. Our findings indicate that RFs are colonised by microbes and that differences in the community structure depend on mineralogical properties and chemical weathering status. We document that RFs are microhabitats with a significant potential to host microbial life in cultivated soils, and thus, could play an important role in biogeochemical cycling and the provision of soil functions in agroecosystems.