High magnetic fields have been of interest for some time in solid-state physics.Recent spectroscopic experiments on highly excited barium atoms and the discovery of very large magnetic fields in white dwarf stars and neutron stars have stimulated renewed interest in the spectroscopy of free atoms in high magnetic fields. We review the various spectroscopic regimes ranging from the low-field Zeeman effect to the high-field Landau regime. We survey calculations of the energy levels of hydrogen, helium and their isoelectronic ions in fields of any strength. We describe observations of the quadratic Zeeman effect in the alkalis and high-field effects in the neutral barium spectrum. Finally, we review recent discoveries of continuum polarization in white dwarf stars, its interpretation in terms of the presence of high magnetic fields, the confirmation of such fields spectroscopically, and some remaining mysteries in these remarkable stellar spectra.