The accurate description of open-shell molecules, in particular of transition metal complexes and clusters, is still an important challenge for quantum chemistry. Although density-functional theory (DFT) is widely applied in this area, the sometimes severe limitations of its currently available approximate realizations often preclude its application as a predictive theory. Here, we review the foundations of DFT applied to open-shell systems, both within the nonrelativistic and the relativistic framework. In particular, we provide an in-depth discussion of the exact theory, with a focus on the role of the spin density and possibilities for targeting specific spin states. It turns out that different options exist for setting up Kohn-Sham DFT schemes for open-shell systems, which imply different definitions of the exchangecorrelation energy functional and lead to different exact conditions on this functional. Finally, we suggest possible directions for future developments.