Cancer cells escape, suppress and exploit the host immune system to sustain themselves, and the tumor microenvironment (TME) actively dampens T cell function by various mechanisms. Over the last years, new immunotherapeutic approaches, such as adoptive chimeric antigen receptor (CAR) T cell therapy and immune checkpoint inhibitors, have been successfully applied for refractory malignancies that could only be treated in a palliative manner previously. Engaging the anti-tumor activity of the immune system, including CAR T cell therapy to target the CD19 B cell antigen, proved to be effective in acute lymphocytic leukemia. In low-grade hematopoietic B cell malignancies, such as chronic lymphocytic leukemia, clinical outcomes have been tempered by cancer-induced T cell dysfunction characterized in part by a state of metabolic lethargy. In multiple myeloma, novel antigens such as BCMA and CD38 are being explored for CAR T cells. In solid cancers, T cell-based immunotherapies have been applied successfully to melanoma and lung cancers, whereas application in e.g., breast cancer lags behind and is modestly effective as yet. The main hurdles for CAR T cell immunotherapy in solid tumors are the lack of suitable antigens, anatomical inaccessibility, and T cell anergy due to immunosuppressive TME. Given the wide range of success and failure of immunotherapies in various cancer types, it is crucial to comprehend the underlying similarities and distinctions in T cell dysfunction. Hence, this review aims at comparing selected, distinct B cell-derived versus solid cancer types and at describing means by which malignant cells and TME might dampen T cell anti-tumor activity, with special focus on immunometabolism. Drawing a meaningful parallel between the efficacy of immunotherapy and the extent of T cell dysfunction will shed light on areas where we can improve immune function to battle cancer.