Single-molecule magnets are among the most promising platforms for achieving molecular-scale data storage and processing. Their magnetisation dynamics are determined by the interplay between electronic and vibrational degrees of freedom, which can couple coherently, leading to complex vibronic dynamics. Building on an ab initio description of the electronic and vibrational Hamiltonians, we formulate a non-perturbative vibronic model of the low-energy magnetic degrees of freedom in a single-molecule magnet, which we benchmark against field-dependent magnetisation measurements. Describing the low-temperature magnetism of the complex in terms of magnetic polarons, we are able to quantify the vibronic contribution to the quantum tunnelling of the magnetisation. Despite collectively enhancing magnetic relaxation, we observe that specific vibrations suppress quantum tunnelling by enhancing the magnetic axiality of the complex. Finally, we discuss how this observation might impact the current paradigm to chemical design of new high-performance single-molecule magnets, promoting vibrations to an active role rather than just regarding them as sources of noise and decoherence.