Cadmium (Cd), a biologically non-essential heavy metal, is widespread in the environment, including the air, water, and soil, and is widely present in foods and quantum dot preparations. Cd enters the body primarily through inhalation and ingestion. Its biological half-life in humans is 10–35 years; therefore, Cd poses long-term health risks. While most studies on Cd toxicity have focused on organ and tissue damage, the immunotoxicity of Cd has drawn increasing attention recently. Cd accumulates in immune cells, modulates the function of the immune system, triggers immunological responses, and leads to diverse health problems. Cd acts as an immunotoxic agent by regulating the activity and apoptosis of immune cells, altering the secretion of immune cytokines, inducing reactive oxygen species (ROS) production and oxidative stress, changing the frequency of T lymphocyte subsets, and altering the production of selective antibodies in immune cells. This review summarizes the immunological toxicity of Cd, elucidates the mechanisms underlying Cd toxicity in terms of innate immunity and adaptive immunity, and discusses potential strategies to alleviate the adverse effects of Cd on the immune system.