The cytolethal distending toxin (CDT) is produced by several Gram-negative pathogenic bacteria. In addition to inflammation, experimental evidences are in favor of a protumoral role of CDT-harboring bacteria such as Escherichia coli, Campylobacter jejuni, or Helicobacter hepaticus. CDT may contribute to cell transformation in vitro and carcinogenesis in mice models, through the genotoxic action of CdtB catalytic subunit. Here, we investigate the mechanism of action by which CDT leads to genetic instability in human cell lines and colorectal organoids from healthy patients’ biopsies. We demonstrate that CDT holotoxin induces a replicative stress dependent on CdtB. The slowing down of DNA replication occurs mainly in late S phase, resulting in the expression of fragile sites and important chromosomic aberrations. These DNA abnormalities induced after CDT treatment are responsible for anaphase bridge formation in mitosis and interphase DNA bridge between daughter cells in G1 phase. Moreover, CDT-genotoxic potential preferentially affects human cycling cells compared to quiescent cells. Finally, the toxin induces nuclear distension associated to DNA damage in proliferating cells of human colorectal organoids, resulting in decreased growth. Our findings thus identify CDT as a bacterial virulence factor targeting proliferating cells, such as human colorectal progenitors or stem cells, inducing replicative stress and genetic instability transmitted to daughter cells that may therefore contribute to carcinogenesis. As some CDT-carrying bacterial strains were detected in patients with colorectal cancer, targeting these bacteria could be a promising therapeutic strategy.