Genome size variation and evolutionary forces behind have been long pursued in flowering plants. The genus Oryza, consisting of approximately 25 wild species and two cultivated rice, harbors eleven extant genome types, six of which are diploid (AA, BB, CC, EE, FF, and GG) and five of which are tetraploid (BBCC, CCDD, HHJJ, HHKK, and KKLL). To obtain the most comprehensive knowledge of genome size variation in the genus Oryza, we performed flow cytometry experiments and estimated genome sizes of 166 accessions belonging to 16 non-AA genome Oryza species. k-mer analyses were followed to verify the experimental results of the two accessions for each species. Our results showed that genome sizes largely varied fourfold in the genus Oryza, ranging from 279 Mb in Oryza brachyantha (FF) to 1,203 Mb in Oryza ridleyi (HHJJ). There was a 2-fold variation (ranging from 570 to 1,203 Mb) in genome size among the tetraploid species, while the diploid species had 3-fold variation, ranging from 279 Mb in Oryza brachyantha (FF) to 905 Mb in Oryza australiensis (EE). The genome sizes of the tetraploid species were not always two times larger than those of the diploid species, and some diploid species even had larger genome sizes than those of tetraploids. Nevertheless, we found that genome sizes of newly formed allotetraploids (BBCC-) were almost equal to totaling genome sizes of their parental progenitors. Our results showed that the species belonging to the same genome types had similar genome sizes, while genome sizes exhibited a gradually decreased trend during the evolutionary process in the clade with AA, BB, CC, and EE genome types. Comparative genomic analyses further showed that the species with different rice genome types may had experienced dissimilar amplification histories of retrotransposons, resulting in remarkably different genome sizes. On the other hand, the closely related rice species may have experienced similar amplification history. We observed that the contents of transposable elements, long terminal repeats (LTR) retrotransposons, and particularly LTR/Gypsy retrotransposons varied largely but were significantly correlated with genome sizes. Therefore, this study demonstrated that LTR retrotransposons act as an active driver of genome size variation in the genus Oryza.