Accumulating evidence suggests that the tumor microenvironment (TME) is involved in disease progression and drug resistance in B cell malignancies, by supporting tumor growth and facilitating the ability of malignant cells to avoid immune recognition. Immunomodulatory drugs (IMiDs) such as lenalidomide have some direct anti-tumor activity, but critically also target various cellular compartments of the TME including T cells, NK cells, and stromal cells, which interfere with pro-tumor signaling while activating anti-tumor immune responses. Lenalidomide has delivered favorable clinical outcomes as a single-agent, and in combination therapy leads to durable responses in chronic lymphocytic leukemia (CLL) and several non-Hodgkin lymphomas (NHLs) including follicular lymphoma (FL), diffuse large B cell lymphoma (DLBCL), and mantle cell lymphoma (MCL). Recently, avadomide, a next generation cereblon E3 ligase modulator (CELMoD), has shown potent anti-tumor and TME immunomodulatory effects, as well as promising clinical efficacy in DLBCL. This review describes how the pleiotropic effects of IMiDs and CELMoDs could make them excellent candidates for combination therapy in the immuno-oncology era—a concept supported by preclinical data, as well as the recent approval of lenalidomide in combination with rituximab for the treatment of relapsed/refractory (R/R) FL.