Context. The young, rapidly rotating Bp star HR 5624 (HD 133880) shows an unusually strong non-sinusoidal variability of its longitudinal magnetic field. This behaviour was previously interpreted as the signature of an exceptionally strong, quadrupole-dominated surface magnetic field geometry. Aims. We studied the magnetic field structure and chemical abundance distributions of HR 5624 with the aim to verify the unusual quadrupolar nature of its magnetic field and to investigate correlations between the field topology and chemical spots. Methods. We analysed high resolution, time series Stokes parameter spectra of HR 5624 with the help of a magnetic Doppler imaging inversion code based on detailed polarised radiative transfer modelling of the line profiles. Results. We refined the stellar parameters, revised the rotational period, and obtained new longitudinal magnetic field measurements. Our magnetic Doppler inversions reveal that the field structure of HR 5624 is considerably simpler and the field strength is much lower than proposed by previous studies. We find a maximum local field strength of 12 kG and a mean field strength of 4 kG, which is about a factor of three weaker than predicted by quadrupolar field models. Our model implies that overall large-scale field topology of HR 5624 is better described as a distorted, asymmetric dipole rather than an axisymmetric quadrupole. The chemical abundance maps of Mg, Si, Ti, Cr, Fe, and Nd obtained in our study are characterised by large-scale, high-contrast abundance patterns. These structures correlate weakly with the magnetic field geometry and, in particular, show no distinct element concentrations in the horizontal field regions predicted by theoretical atomic diffusion calculations. Conclusions. We conclude that the surface magnetic field topology of HR 5624 is not as unusual as previously proposed. Considering these results together with other recent magnetic mapping analyses of early-type stars suggests that predominantly quadrupolar magnetic field topologies, invoked to be present in a significant number of stars, probably do not exist in real stars. This finding agrees with an outcome of the MHD simulations of fossil field evolution in stably stratified stellar interiors.