Chromosomal instability (CIN) is associated with many human diseases, including neurodevelopmental or neurodegenerative conditions, age-related disorders and cancer, and is a key driver for disease initiation and progression. A major source of structural chromosome instability (s-CIN) leading to structural chromosome aberrations is “replication stress”, a condition in which stalled or slowly progressing replication forks interfere with timely and error-free completion of the S phase. On the other hand, mitotic errors that result in chromosome mis-segregation are the cause of numerical chromosome instability (n-CIN) and aneuploidy. In this review, we will discuss recent evidence showing that these two forms of chromosomal instability can be mechanistically interlinked. We first summarize how replication stress causes structural and numerical CIN, focusing on mechanisms such as mitotic rescue of replication stress (MRRS) and centriole disengagement, which prevent or contribute to specific types of structural chromosome aberrations and segregation errors. We describe the main outcomes of segregation errors and how micronucleation and aneuploidy can be the key stimuli promoting inflammation, senescence, or chromothripsis. At the end, we discuss how CIN can reduce cellular fitness and may behave as an anticancer barrier in noncancerous cells or precancerous lesions, whereas it fuels genomic instability in the context of cancer, and how our current knowledge may be exploited for developing cancer therapies.