Cancer evolution is explained by the accumulation of driver mutations and subsequent positive selection by acquired growth advantages, like Darwin's evolution theory. However, whether the negative selection of cells that have lost malignant properties contributes to cancer progression has not yet been fully investigated. Using intestinal metastatic tumor‐derived organoids carrying Apc, Kras, Tgfbr2, and Trp53 quadruple mutations, we demonstrate here that approximately 30% of subclones of the organoids show loss of metastatic ability to the liver while keeping the driver mutations and oncogenic pathways. Notably, highly metastatic subclones also showed a gradual loss of metastatic ability during further passages. Such non‐metastatic subclones revealed significantly decreased survival and proliferation ability in Matrigel and collagen gel culture conditions, which may cause elimination from the tumor tissues in vivo. RNA sequencing indicated that stemness‐related genes, including Lgr5 and Myb, were significantly downregulated in non‐metastatic subclones as well as subclones that lost metastatic ability during additional passages. Furthermore, a CGH analysis showed that non‐metastatic subclones were derived from a minor population of parental organoid cells. These results indicate that metastatic ability is continuously lost with decreased stem cell property in certain subpopulations of malignant tumors, and such subpopulations are eliminated by negative selection. Therefore, it is possible that cancer evolution is regulated not only by positive selection but also by negative selection. The mechanism underlying the loss of metastatic ability will be important for the future development of therapeutic strategies against metastasis.