Cavity mediated modification of material properties and phenomena is a novel research field largely motivated by the experimental advances in strong light-matter interactions. Despite this progress, exact solutions for extended systems strongly coupled to the photon field are not available, and both theory and experiments rely mainly on finite-system models. Therefore a paradigmatic example of an exactly solvable extended system in a cavity becomes highly desireable. To fill this gap we revisit Sommerfeld's theory of the free electron gas in cavity quantum electrodynamics (QED). We solve this system analytically in the long-wavelength limit for an arbitrary number of non-interacting electrons. This exact solution allows us to consider the thermodynamic limit for the electrons and to demonstrate several new features: The four fundamental response functions (matter-matter, photon-matter, matter-photon and photon-photon) are proportional to each other and all exhibit plasmon-polariton excitations, which modify the conductive properties of the electron gas. In contrast to finite systems, no ground state exists if the diamagentic A 2 term is omitted. Further, we show that the thermodynamic limit can be performed only if we take simultaneously the size of the matter system and the amount of photon modes to infinity. To accomplish this we introduce an effective theory for the photon field. The effective coupling and the ultraviolet behavior of the theory are well-defined, and the continuum of modes leads to a renormalization of the electron mass, which in contrast to the usual single-particle renormalization depends on the full electron density. Lastly, we show how the matter-modified photon field leads to a repulsive Casimir force and how the continuum of modes introduces dissipation into the light-matter system. Several of the presented findings should be experimentally accessible.