HIV infection and antiretroviral therapy (ART) treatment exert diverse effects on adipocytes and stromal-vascular fraction cells, leading to changes in adipose tissue quantity, distribution, and energy storage. A HIV-associated lipodystrophic condition was recognized early in the epidemic, characterized by clinically apparent changes in subcutaneous, visceral, and dorsocervical adipose depots. Underlying these changes is altered adipose tissue morphology and expression of genes central to adipocyte maturation, regulation, metabolism, and cytokine signaling. HIV viral proteins persist in circulation and locally within adipose tissue despite suppression of plasma viremia on ART, and exposure to these proteins impairs preadipocyte maturation and reduces adipocyte expression of peroxisome proliferator-activated receptor gamma (PPAR-γ) and other genes involved in cell regulation. Several early nucleoside reverse transcriptase inhibitor and protease inhibitor antiretroviral drugs demonstrated substantial adipocyte toxicity, including reduced mitochondrial DNA content and respiratory chain enzymes, reduced PPAR-γ and other regulatory gene expression, and increased pro-inflammatory cytokine production. Newer-generation agents, such as integrase inhibitors, appear to have fewer adverse effects. HIV infection also alters the balance of CD4+ and CD8+ T cells in adipose tissue, with effects on macrophage activation and local inflammation, while the presence of latently-infected CD4+ T cells in adipose tissue may constitute a protected viral reservoir. This review provides a synthesis of the literature on how HIV virus, ART treatment, and host characteristics interact to affect adipose tissue distribution, immunology, and contribution to metabolic health, and adipocyte maturation, cellular regulation, and energy storage.