In sequential multiscale molecular dynamics simulations, which advantageously combine the increased sampling and dynamics at coarse-grained resolution with the higher accuracy ofatomistic simulations, the resolution is altered over time. While coarse-graining is straightforward, the reintroduction of the atomistic detail is a non-trivial process called backmapping. Here, we present ART-SM, a fragment-based machine learning backmapping framework that learns the Boltzmann distribution from atomistic data to switch from coarse-grained to atomistic resolution seamlessly. ART-SM requires minimal user input and goes beyond state-of-the-art fragment-based approaches by selecting from multiple conformations per fragment to simultaneously reflect the coarse-grained structure and the Boltzmann distribution. Additionally, we introduce a novel refinement step to connect individual fragments via optimization of specific bonds, angles, and dihedral angles in the backmapping process. We demonstrate that our algorithm accurately restores the atomistic bond length, angle, and dihedral angle distributions for various small molecules of up to three Martini coarse-grained beads and that the resulting high-resolution structures are representative of the original coarse-grained conformations. Moreover, the reconstruction of the TIP3P water model is fast and robust, and we illustrate that ART-SM can be, in principle, applied to larger molecules as well, indicating its potential extension to more complex molecules like lipids, proteins, and macromolecules in the future.