We consider a hybrid quantum many-body system formed by a vibrational mode of a nanomembrane, which interacts optomechanically with light in a cavity, and an ultracold atom gas in the optical lattice of the out-coupled light. The adiabatic elimination of the light field yields an effective Hamiltonian which reveals a competition between the force localizing the atoms and the membrane displacement. At a critical atom-membrane interaction, we find a nonequilibrium quantum phase transition from a localized symmetric state of the atom cloud to a shifted symmetry-broken state, the energy of the lowest collective excitation vanishes, and a strong atom-membrane entanglement arises. The effect occurs when the atoms and the membrane are non-resonantly coupled.Hybrid quantum systems combine complementary fields of physics, such as solid-state physics, quantum optics and atom physics, in one set-up. Recently, a hybrid atom-optomechanical system [1] has been realized experimentally [2] in which a single mechanical mode of a nanomembrane in an optical cavity is optically coupled to a far distant cloud of cold 87 Rb atoms residing in the optical potential of the out-coupled standing wave of the cavity light. When displaced, the membrane experiences the radiation pressure force of the cavity light, and in the bad-cavity limit, the field follows the membrane displacement adiabatically. This modulates the light phase which leads to a shaking of the atom gas in the lattice. The nanomechanical motion of the membrane then couples non-resonantly to the collective motion of the atoms. The aim is twofold [1][2][3][4][5][6][7]: The gas can cool the nanomembrane, and, emergent phenomena of the correlated quantum many-body system are of interest [8][9][10][11][12][13][14][15][16][17].State-of-the-art optomechanics [3-6] is nowadays able to realize optical feedback cooling [18,19] of the mechanical oscillator to its quantum-mechanical ground state [20,21]. Yet, the resolved sideband limit allows ground-state cooling only if the oscillator frequency exceeds the photon loss rate in the cavity [22][23][24]. Hence, cooling a macroscopic low-frequency nanomembrane close to its ground state is so far not possible. One promising alternative [1,7] is to utilize an ultracold atom gas, which has been demonstrated recently [2] by sympathetic cooling down to 650 mK. Current investigations aim to a coherent state transfer of robust quantum entanglement [15].Apart from cooling the nanomembrane, an interesting fundamental feature is the collective quantum-many body behavior of the hybrid system. For instance, the atom-atom interaction can in principle be coherently modulated by the back-action of the cavity light on the nanooscillator. By this, a long-range interaction emerges which resembles that of a dipolar Bose-Einstein condensate (BEC) [25]. In fact, a simpler hybrid quantum many-body system has also been implemented in the form of a BEC in an optical lattice inside a transversely pumped optical cavity. A Dicke quantum phase transition between a n...