We first propose and study a quantum toy model of black hole dynamics. The model is unitary, displays quantum thermalization, and the Hamiltonian couples every oscillator with every other, a feature intended to emulate the color sector physics of large-N matrix models. Considering out of equilibrium initial states, we analytically compute the time evolution of every correlator of the theory and of the entanglement entropies, allowing a proper discussion of global thermalization/scrambling of information through the entire system. Microscopic non-locality causes factorization of reduced density matrices, and entanglement just depends on the time evolution of occupation densities. In the second part of the article, we show how the gained intuition extends to large-N matrix models, where we provide a gauge invariant entanglement entropy for 'generalized free fields', again depending solely on the quasinormal frequencies. The results challenge the fast scrambling conjecture and point to a natural scenario for the emergence of the so-called brick wall or stretched horizon. Finally, peculiarities of these models in regards to the thermodynamic limit and the information paradox are highlighted.