The tumor microenvironment (TME) is composed of multiple infiltrating host cells (e.g., endothelial cells, fibroblasts, lymphocytes, and myeloid cells), extracellular matrix, and various secreted or cell membrane-presented molecules. Group 1 innate lymphoid cells (ILCs), which includes natural killer (NK) cells and ILC1, contribute to protecting the host against cancer and infection. Both subsets are able to quickly produce cytokines such as interferon gamma (IFN-γ), chemokines, and other growth factors in response to activating signals. However, the TME provides many molecules that can prevent the potential effector function of these cells, thereby protecting the tumor. For example, TME-derived tumor growth factor (TGF)-β and associated members of the superfamily downregulate NK cell cytotoxicity, cytokine secretion, metabolism, proliferation, and induce effector NK cells to upregulate ILC1-like characteristics. In concert, a family of carbohydrate-binding proteins called galectins, which can be produced by different cells composing the TME, can downregulate NK cell function. Matrix metalloproteinase (MMP) and a disintegrin and metalloproteinase (ADAM) are also enzymes that can remodel the extracellular matrix and shred receptors from the tumor cell surface, impairing the activation of NK cells and leading to less effective effector functions. Gaining a better understanding of the characteristics of the TME and its associated factors, such as infiltrating cells and extracellular matrix, could lead to tailoring of new personalized immunotherapy approaches. This review provides an overview of our current knowledge on the impact of the TME and extracellular matrix-associated components on differentiation, impairment, and function of NK cells.