Most theoretical studies for correlated light-matter systems are performed within the longwavelength limit, i.e., the electromagnetic field is assumed to be spatially uniform. In this limit the so-called length-gauge transformation for a fully quantized light-matter system gives rise to a dipole self-energy term in the Hamiltonian, i.e., a harmonic potential of the total dipole matter moment. In practice this term is often discarded as it is assumed to be subsumed in the kinetic energy term. In this work we show the necessity of the dipole self-energy term. First and foremost, without it the light-matter system in the long-wavelength limit does not have a ground-state, i.e., the combined light-matter system is unstable. Further, the mixing of matter and photon degrees of freedom due to the length-gauge transformation, which also changes the representation of the translation operator for matter, gives rise to the Maxwell equations in matter and the omittance of the dipole self-energy leads to a violation of these equations. Specifically we show that without the dipole self-energy the so-called "depolarization shift" is not properly described. Finally we show that this term also arises if we perform the semi-classical limit after the length-gauge transformation. In contrast to the standard approach where the semi-classical limit is performed before the length-gauge transformation, the resulting Hamiltonian is bounded from below and thus supports ground states. This is very important for practical calculations and for density-functional variational implementations of the non-relativistic QED formalism. For example, the existence of a combined light-matter ground state allows to calculate the Stark shift non-perturbatively. *