We study the low temperature physics of an ultracold atomic gas in the potential formed inside a pumped optical resonator. Here, the height of the cavity potential, and hence the quantum state of the gas, depends not only on the pump parameters, but also on the atomic density through a dynamical ac-Stark shift of the cavity resonance. We derive the Bose-Hubbard model in one dimension and use the strong coupling expansion to determine the parameter regime in which the system is in the Mott-insulator state. We predict the existence of overlapping, competing Mott-insulator states, and bistable behavior in the vicinity of the shifted cavity resonance, controlled by the pump parameters. Outside these parameter regions, the state of the system is in most cases superfluid. DOI: 10.1103/PhysRevLett.100.050401 PACS numbers: 05.30.Jp, 03.75.Hh, 37.10.Jk, 67.90.+z Ultracold atomic gases in optical lattices offer the unprecedented and unique possibility to study paradigmatic systems of quantum many-body physics [1,2]. These systems allow one to realize various versions of Hubbard models [3], a prominent example of which is the BoseHubbard model [4], exhibiting the superfluid (SF)-Mottinsulator (MI) quantum phase transition [5]. The realization of the Bose-Hubbard model with ultracold atoms has been proposed in the seminal theoretical work in Ref. [6] and has been demonstrated in the milestone experiments in Ref. [7]. Several aspects and modifications of the SF-MI quantum phase transition (or crossover [8]) are the objects of intense studies [2].Optical lattices in free space are not affected by the presence of the atoms. This scenario is, however, strongly modified when the atoms move in the optical potential which is formed inside a pumped resonator: Here, the atoms interact with the cavity mode while the cavity field, determining the optical lattice, may critically depend on the density of the atoms [9,10]. Several recent studies address cavity quantum electrodynamics (CQED) with cold atoms. CQED techniques were used to measure pair correlations in the atom laser [11] and have been proposed for characterizing quantum states of ultracold matter [12]. Self-organization of atoms in transversally pumped cavities was observed in [13] and was theoretically described in [14]. Bragg scattering of atomic structures inside optical resonators has been investigated in [15]. Most recently, Bose-Einstein condensed atoms have been loaded inside cavities [16]. This experimental progress calls for theoretical development of CQED combined with many-body physics.In this Letter we determine the ground state of ultracold atomic gases in the optical lattice of a cavity. The cavity is driven by a laser, and the atoms shift the cavity resonance, thus affecting the intracavity field amplitude, which in turn determines the depth of the cavity potential and the ground state of the atomic gas itself. The problem is hence highly nonlinear, as the optical lattice and the state of the atoms have to be evaluated in a self-consistent way. The derivat...