Liposomes hold great potential as gene and drug delivery vehicles due to their biocompatibility and modular properties, coupled with the major advantage of attenuating the risk of systemic toxicity from the encapsulated therapeutic agent. Decades of research have been dedicated to studying and optimizing liposomal formulations for a variety of medical applications, ranging from cancer therapeutics to analgesics. Some effort has also been made to elucidate the toxicities and immune responses that these drug formulations may elicit. Notably, intravenously injected liposomes can interact with plasma proteins, leading to opsonization, thereby altering the healthy cells they come into contact with during circulation and removal. Additionally, due to the pharmacokinetics of liposomes in circulation, drugs can end up sequestered in organs of the mononuclear phagocyte system, affecting liver and spleen function. Importantly, liposomal agents can also stimulate or suppress the immune system depending on their physiochemical properties, such as size, lipid composition, pegylation, and surface charge. Despite the surge in the clinical use of liposomal agents since 1995, there are still several drawbacks that limit their range of applications. This review presents a focused analysis of these limitations, with an emphasis on toxicity to healthy tissues and unfavorable immune responses, to shed light on key considerations that should be factored into the design and clinical use of liposomal formulations.