MUSE observations of NGC 5813 reveal a complex structure in the velocity dispersion map, previously hinted by SAURON observations. The structure is reminiscent of velocity dispersion maps of galaxies comprising two counter-rotating discs, and may explain the existence of the kinematically distinct core (KDC). Further evidence for two counter-rotating components comes from the analysis of the higher moments of the stellar line-of-sight velocity distributions and fitting MUSE spectra with two separate Gaussian line-of-sight velocity distributions. The emission-line kinematics show evidence of being linked to the present cooling flows and the buoyant cavities seen in Xrays. We detect ionised gas in a nuclear disc-like structure, oriented like the KDC, which is, however, not directly related to the KDC. We build an axisymmetric Schwarzschild dynamical model, which shows that the MUSE kinematics can be reproduced well with two counter-rotating orbit families, characterised by relatively low angular momentum components, but clearly separated in integral phase space and with radially varying contributions. The model indicates that the counter-rotating components in NGC 5813 are not thin discs, but dynamically hot structures. Our findings give further evidence that KDCs in massive galaxies should not necessarily be considered as structurally or dynamically decoupled regions, but as the outcomes of the mixing of different orbital families, where the balance in the distribution of mass of the orbital families is crucial. We discuss the formation of the KDC in NGC 5813 within the framework of gas accretion, binary mergers and formation of turbulent thick discs from cold streams at high redshift.