The Dicke model with a weak dissipation channel is realized by coupling a Bose-Einstein condensate to an optical cavity with ultranarrow bandwidth. We explore the dynamical critical properties of the Hepp-Lieb-Dicke phase transition by performing quenches across the phase boundary. We observe hysteresis in the transition between a homogeneous phase and a self-organized collective phase with an enclosed loop area showing power-law scaling with respect to the quench time, which suggests an interpretation within a general framework introduced by Kibble and Zurek. The observed hysteretic dynamics is well reproduced by numerically solving the mean-field equation derived from a generalized Dicke Hamiltonian. Our work promotes the understanding of nonequilibrium physics in open many-body systems with infinite range interactions.dynamical phase transition | critical behavior | Dicke model | quantum gas | cavity QED A lthough equilibrium phases in quantum many-body systems have been explored for a long time with great success, nonequilibrium phenomena in such systems are far less well understood (1). A paradigm for exploring nonequilibrium dynamics is the quench scenario, where a system parameter is subjected to a sudden change between two values associated with different equilibrium phases. Quantum degenerate atomic gases with their unique degree of control are particularly adapted for experimental quench studies (2, 3). For isolated quantum manybody systems a wealth of theoretical and experimental investigations of quench dynamics has appeared recently (4-11). A natural extension of such studies is to consider driven open systems, where dynamical equilibrium states can arise via a competition between dissipation and driving, and nonequilibrium transitions between such phases can occur as a function of some external control parameter (12-15). A nearly ideal experimental platform for this endeavor are quantum degenerate atomic gases subjected to optical high-finesse cavities, where the usual extensive control in cold gas systems can be combined with a precisely engineered coupling to the external bath of vacuum radiation modes (16).Here, we study a dynamical phase transition in the open Dicke model emulated in an atom-cavity system prepared near zero temperature. The Dicke model is a paradigmatic scenario of quantum many-body physics, still subject to intensive research despite a history more than half a century long (17-28). It describes the interaction of N two-level atoms with a common mode of the electromagnetic radiation field. Hepp and Lieb already pointed out in the 1970s that upon varying the coupling strength, this model possesses a second-order equilibrium quantum phase transition between a homogeneous phase, in which each atom interacts separately with the radiation mode, and a collective phase in which all atomic dipoles align to form a macroscopic dipole moment (19,22). It has been early suspected that the critical properties of the externally pumped open Dicke model should give rise to nonlinear hysteretic ...