In colorectal cancer, oncogenic mutations transform a hierarchically organized and homeostatic epithelium into invasive cancer tissue. To define differences in cellular composition between the normal colon and colorectal cancer, and to map potential cellular interactions between tumor cells and their microenvironment, we profiled transcriptomes of >50,000 single cells from tumors and matched normal tissues of eight colorectal cancer patients. We find that tumor formation is accompanied by changes in epithelial, immune and stromal cell compartments in all patients. In the epithelium, we identify a continuum of five tumor-specific stem cell and progenitor-like populations, and persistent multilineage differentiation. We find multiple stromal and immune cell types to be consistently expanded in tumor compared to the normal colon, including cancer-associated fibroblasts, pericytes, monocytes, macrophages and a subset of T cells. We identify epithelial tumor cells and cancer-associated fibroblasts as relevant for assigning colorectal cancer consensus molecular subtypes. Our survey of growth factors in the tumor microenvironment identifies cell types responsible for increased paracrine EGFR, MET and TGF-β signaling in tumor tissue compared to the normal colon.We show that matched colorectal cancer organoids retain cell type heterogeneity, allowing to define a distinct differentiation trajectory encompassing stem and progenitor-like tumor cells. In summary, our single-cell analyses provide insights into cell types and signals shaping colorectal cancer cell plasticity.