Pithoviruses are amoeba-infecting giant viruses possessing the largest viral particles known so far. Since the discovery of Pithovirus sibericum, recovered from a 30,000-y-old permafrost sample, other pithoviruses, and related cedratviruses, were isolated from various terrestrial and aquatic samples. Here we report the isolation and genome sequencing of two strains of Pithoviridae from soil samples, in addition to three other recent isolates. Using the 12 available genome sequences, we conducted a thorough comparative genomics study of the Pithoviridae family to decipher the organization and evolution of their genomes. Our study reveals a non-uniform genome organization in two main regions: one concentrating core genes, and another gene duplications. We also found that Pithoviridae genomes are more conservative than other families of giant viruses, with a low and stable proportion (5% to 7%) of genes originating from horizontal transfers. Genome size variation within the family is mainly due to variations in gene duplication rates (from 14% to 28%) and massive invasion by miniature inverted-repeats transposable elements (MITEs). While these repeated elements are absent from cedratviruses, repeat-rich regions cover as much as a quarter of the pithoviruses genomes. These regions, identified using a dedicated pipeline, are hotspots of mutations, gene capture events and genomic rearrangements, that likely contribute to their evolution.