Human tumors are increasingly being described as a complex “ecosystem”, that includes many different cell types, secreted growth factors, extracellular matrix (ECM) components, and microvessels, that altogether create the tumor microenvironment (TME). Within the TME, epithelial cancer cells control the function of surrounding stromal cells and the non-cellular ECM components in an intricate orchestra of signaling networks specifically designed for cancer cells to exploit surrounding cells for their own benefit. Tumor-derived extracellular vesicles (EVs) released into the tumor microenvironment are essential mediators in the reprogramming of surrounding stromal cells, which include cancer-associated fibroblasts (CAFs), tumor-associated macrophages (TAMs), tumor-infiltrating lymphocytes (TILs), and tumor endothelial cells (TECs), which are responsible for the promotion of neo-angiogenesis, immune cell evasion, and invasion which are essential for cancer progression. Perhaps most importantly, tumor-derived EVs play critical roles in the metastatic dissemination of tumor cells through their two-fold role in initiating cancer cell invasion and the establishment of the pre-metastatic niche, both of which are vital for tumor cell migration, homing, and colonization at secondary tumor sites. This review discusses extracellular vesicle trafficking within the tumor microenvironment and pre-metastatic niche formation, focusing on the complex role that EVs play in orchestrating cancer-to-stromal cell communication in order to promote the metastatic dissemination of cancer cells.