Bacteria generally live in species-rich communities, such as the gut microbiota. Yet, little is known about bacterial evolution in natural ecosystems. Here, we followed the long-term evolution of commensal Escherichia coli in the mouse gut. We observe the emergence of polymorphism for mutation rate, ranging from wild-type levels to 1000-fold higher. By combining experiments, whole-genome sequencing and in silico simulations, we identify the molecular causes and evolutionary conditions that allow these hypermutators to emerge and coexist within a complex microbiota. The hypermutator phenotype is caused by mutations in DNA polymerase III, which increase mutation rate by~1000-fold (a mutation in the proofreading subunit) and stabilize hypermutator fitness (mutations in the catalytic subunit). The strong mutation rate variation persists for >1000 generations, with coexistence between lineages carrying 4 to >600 mutations. This in vivo molecular evolution pattern is consistent with deleterious mutations of~0.01-0.001% fitness effects, 100 to 1000-fold lower than current in vitro estimates. Despite large numbers of deleterious mutations, we identify multiple beneficial mutations that do not reach fixation over long periods of time. This indicates that the dynamics of beneficial mutations are not shaped by constant positive Darwinian selection but by processes leading to negative frequency-dependent or temporally fluctuating selection. Thus, microbial evolution in the gut is likely characterized by partial sweeps of beneficial mutations combined with hitchhiking of very slightly deleterious mutations, which take a long time to be purged but impose a very weak mutational load. These results are consistent with the pattern of genetic polymorphism that is emerging from metagenomics studies of the human gut microbiota, suggesting that we identified key evolutionary processes shaping the genetic composition of this community.