We develop a three-dimensional multispecies mathematical model to simulate the growth of colon cancer organoids containing stem, progenitor and terminally differentiated cells, as a model of early (prevascular) tumor growth. Stem cells (SCs) secrete short-range self-renewal promoters (e.g., Wnt) and their long-range inhibitors (e.g., Dkk) and proliferate slowly. Committed progenitor (CP) cells proliferate more rapidly and differentiate to produce post-mitotic terminally differentiated cells that release differentiation promoters, forming negative feedback loops on SC and CP self-renewal. We demonstrate that SCs play a central role in normal and cancer colon organoids. Spatial patterning of the SC self-renewal promoter gives rise to SC clusters, which mimic stem cell niches, around the organoid surface, and drive the development of invasive fingers. We also study the effects of externally applied signaling factors. Applying bone morphogenic proteins, which inhibit SC and CP self-renewal, reduces invasiveness and organoid size. Applying hepatocyte growth factor, which enhances SC self-renewal, produces larger sizes and enhances finger development at low concentrations but suppresses fingers at high concentrations. These results are consistent with recent experiments on colon organoids. Because many cancers are hierarchically organized and are subject to feedback regulation similar to that in normal tissues, our results suggest that in cancer, control of cancer stem cell self-renewal should influence the size and shape in similar ways, thereby opening the door to novel therapies.