Despite a growing wealth of available molecular data, the growth of tumors, invasion of tumors into healthy tissue, and response of tumors to therapies are still poorly understood. Although genetic mutations are in general the first step in the development of a cancer, for the mutated cell to persist in a tissue, it must compete against the other, healthy or diseased cells, for example by becoming more motile, adhesive, or multiplying faster. Thus, the cellular phenotype determines the success of a cancer cell in competition with its neighbors, irrespective of the genetic mutations or physiological alterations that gave rise to the altered phenotype. What phenotypes can make a cell “successful” in an environment of healthy and cancerous cells, and how? A widely used tool for getting more insight into that question is cell-based modeling. Cell-based models constitute a class of computational, agent-based models that mimic biophysical and molecular interactions between cells. One of the most widely used cell-based modeling formalisms is the cellular Potts model (CPM), a lattice-based, multi particle cell-based modeling approach. The CPM has become a popular and accessible method for modeling mechanisms of multicellular processes including cell sorting, gastrulation, or angiogenesis. The CPM accounts for biophysical cellular properties, including cell proliferation, cell motility, and cell adhesion, which play a key role in cancer. Multiscale models are constructed by extending the agents with intracellular processes including metabolism, growth, and signaling. Here we review the use of the CPM for modeling tumor growth, tumor invasion, and tumor progression. We argue that the accessibility and flexibility of the CPM, and its accurate, yet coarse-grained and computationally efficient representation of cell and tissue biophysics, make the CPM the method of choice for modeling cellular processes in tumor development.