Cancer cachexia (CC) is a multifactorial syndrome in patients with advanced cancer characterized by weight loss via skeletal-muscle and adipose-tissue atrophy, catabolic activity, and systemic inflammation. CC is correlated with functional impairment, reduced therapeutic responsiveness, and poor prognosis, and is a major cause of death in cancer patients. In colorectal cancer (CRC), cachexia affects around 50–61% of patients, but remains overlooked, understudied, and uncured. The mechanisms driving CC are not fully understood but are related, at least in part, to the local and systemic immune response to the tumor. Accumulating evidence demonstrates a significant role of tumor microenvironment (TME) cells (e.g., macrophages, neutrophils, and fibroblasts) in both cancer progression and tumor-induced cachexia, through the production of multiple procachectic factors. The most important role in CRC-associated cachexia is played by pro-inflammatory cytokines, including the tumor necrosis factor α (TNFα), originally known as cachectin, Interleukin (IL)-1, IL-6, and certain chemokines (e.g., IL-8). Heterogeneous CRC cells themselves also produce numerous cytokines (including chemokines), as well as novel factors called “cachexokines”. The tumor microenvironment (TME) contributes to systemic inflammation and increased oxidative stress and fibrosis. This review summarizes the current knowledge on the role of TME cellular components in CRC-associated cachexia, as well as discusses the potential role of selected mediators secreted by colorectal cancer cells in cooperation with tumor-associated immune and non-immune cells of tumor microenvironment in inducing or potentiating cancer cachexia. This knowledge serves to aid the understanding of the mechanisms of this process, as well as prevent its consequences.