Osteons, the main organizational components of human compact bone, are cylindrical structures composed of layers of mineralized collagen fibrils, called lamellae. These lamellae have different orientations, different degrees of organization and different degrees of mineralization where the intrafibrillar and extrafibrillar mineral is intergrown into one continuous network of oriented crystals. While cellular activity is clearly the source of the organic matrix, recent in vitro studies call into question whether the cells are also involved in matrix mineralization, and suggest that this process could be simply driven by the physicochemical conditions in the extracellular matrix. Through the remineralization of demineralized bone matrix, we demonstrate the complete multiscale reconstruction of the 3D structure and composition of the osteon without cellular involvement. We then explore this cell-free in vitro system as a realistic, functional model for the in situ investigation of matrix-controlled mineralization processes. Using a combination of Raman and electron microscopy we show that glycosaminoglycans play a more prominent role than generally assumed in the matrix-mineral interactions, which determine how fast, were, and in which form the mineral is deposited. Our experiments also show that the organization of the collagen is in part a result of its interaction with the developing mineral.