Ultrastrong coupling between light and matter has, in the past decade, transitioned from theoretical idea to experimental reality. It is a new regime of quantum light-matter interaction, going beyond weak and strong coupling to make the coupling strength comparable to the transition frequencies in the system. The achievement of weak and strong coupling has led to increased control of quantum systems and applications like lasers, quantum sensing, and quantum information processing. Here we review the theory of quantum systems with ultrastrong coupling, which includes entangled ground states with virtual excitations, new avenues for nonlinear optics, and connections to several important physical models. We also review the multitude of experimental setups, including superconducting circuits, organic molecules, semiconductor polaritons, and optomechanics, that now have achieved ultrastrong coupling. We then discuss the many potential applications that these achievements enable in physics and chemistry.