The revolutionary discovery of resting state networks radically shifted the focus from the role of local regions in cognitive tasks to the ongoing spontaneous dynamics in global networks. Yet, there is a growing realisation that these resting state networks could be a bit like the shadow tracings in Plato’s famous cave, perhaps mere epiphenomena of an underlying hidden space from where these shadows emanate. Here we used deep variational auto-encoders to extract manifolds of low dimensionality from whole-brain dynamics measured with functional magnetic resonance imaging (fMRI). Crucially, we constructed the first dynamical model of the low dimensional manifold modes, i.e., networks of nodes using non-linear oscillators coupled with the effective functional connectivity, taking into account the level of non-equilibrium dynamics quantified by the non-reversibility of the signals. Irrespective of parcellation size, we found an optimal number of roughly ten manifold modes to best describe the whole-brain activity. Importantly, compared to traditional whole-brain modelling using all the nodes in a parcellation, we obtained better results for resting and task activity by modelling the dynamics of the coupled manifold modes. These findings show the key causal role of manifolds as a fundamental organising principle of brain function at the whole-brain scale, providing evidence that networks of brain regions rather than individual brain regions are the key computational engines of the brain.