Rhizoliths, that is, roots fossilized by secondary carbonates, have been known for ages and are increasingly used for paleoenvironmental reconstructions. However, knowledge about their formation mechanisms remains limited. This study reports the mineralogical and chemical characterization of rhizoliths at different stages of mineralization and fossilization in the Late Pleistocene loess–paleosol sequence of Nussloch (SW Germany). Scanning electron microscopy coupled with elemental mapping and 13C solid‐state nuclear magnetic resonance were used to concomitantly characterize the mineral and organic matter of the rhizoliths. These joint analyses showed for the first time that large rhizoliths are not necessarily remains of single large roots but consist of numerous microrhizoliths as remains of fine roots, formed mainly by calcium carbonates with only low amounts of Mg and Si. They further revealed that the precipitation of secondary carbonates occurs not only around, but also within the plant root and that fossilization leads to the selective preservation of recalcitrant root biopolymers—lignin and suberin. The precipitation of secondary carbonates was observed to occur first around fine roots, the epidermis acting as a first barrier, and then within the root, within the cortex cells, and even sometimes around the phloem and within the xylem. This study suggests that the calcification of plant roots starts during the lifetime of the plant and continues after its death. This has to be systematically investigated to understand the stratigraphic context before using (micro)rhizoliths for paleoenvironmental reconstructions in terrestrial sediments.