Hybrid systems of laser-cooled trapped ions and ultracold atoms combined in a single experimental setup have recently emerged as a new platform for fundamental research in quantum physics. This paper reviews the theoretical and experimental progress in research on cold hybrid ion-atom systems which aim to combine the best features of the two well-established fields. We provide a broad overview of the theoretical description of ion-atom mixtures and their applications, and report on advances in experiments with ions trapped in Paul or dipole traps overlapped with a cloud of cold atoms, and with ions directly produced in a Bose-Einstein condensate. We start with microscopic models describing the electronic structure, interactions, and collisional physics of ion-atom systems at low and ultralow temperatures, including radiative and non-radiative charge transfer processes and their control with magnetically tunable Feshbach resonances. Then we describe the relevant experimental techniques and the intrinsic properties of hybrid systems. In particular, we discuss the impact of the micromotion of ions in Paul traps on ion-atom hybrid systems. Next, we review recent proposals for using ions immersed in ultracold gases for studying cold collisions, chemistry, many-body physics, quantum simulation, and quantum computation and their experimental realizations. In the last part we focus on the formation of molecular ions via spontaneous radiative association, photoassociation, magnetoassociation, and sympathetic cooling. We discuss applications and prospects of cold molecular ions for cold controlled chemistry and precision spectroscopy.