Genetic instability is an important characteristic of cancer. While most cancers develop genetic instability at some stage of their progression, sometimes a temporary rise of instability is followed by the return to a relatively stable genome. Neither the reasons for these dynamics, nor, more generally, the role of instability in tumor progression, are well understood. In this paper we develop a class of mathematical models to study the evolutionary competition dynamics among different sub-populations in a heterogeneous tumor. We observe that despite the complexity of this multi-component and multi-process system, there is only a small number of scenarios expected in the context of the evolution of instability. If the penalty incurred by unstable cells (the decrease in the growth due to deleterious mutations) is high compared with the gain (the production rate of advantageous mutations), then instability does not evolve. In the opposite case, instability evolves and comes to dominate the system. In the intermediate parameter regime, instability is generated but later gives way to stable clones. Moreover, the model also informs us of the patterns of instability for cancer lineages corresponding to different stages of progression. It is predicted that mutations causing instability are merely “passengers” in tumors that have undergone only a small number of malignant mutations. Further down the path of carcinogenesis, however, unstable cells are more likely to give rise to the winning clonal wave that takes over the tumor and carries the evolution forward, thus conferring a causal role of the instability in such cases. Further, each individual clonal wave (i.e. cells harboring a fixed number of malignant driver mutations) experiences its own evolutionary history. It can fall under one of three types of temporal behavior: stable throughout, unstable to stable, or unstable throughout. Which scenario is realized depends on the subtle (but predictable) interplay among mutation rates and the death toll associated with the instability. The modeling approach provided here sheds light onto important aspects of the evolutionary dynamics of instability, which may be relevant to treatment scenarios that target instability or damage repair.

No abstract