Viral fossils in rice genomes are a best entity to understand ancient pararetrovirus activities through host plant history because of our advanced knowledge of the genomes and evolutionary history with rice and its related species. Here, we explored organization, geographic origins and genealogy of rice pararetroviruses, which were turned into endogenous rice tungro bacilliform virus-like (eRTBVL) sequences. About 300 eRTBVL sequences from three representative rice genomes were clearly classified into six families. Most of the endogenization events of the eRTBVLs were initiated before differentiation of the rice progenitor (> 160,000 years ago). We successfully followed the genealogy of old relic viruses during rice speciation, and inferred the geographical origins for these viruses. Possible virus genomic sequences were explained mostly by recombinations between different virus families. Interestingly, we discovered that only a few recombination events among the numerous occasions had determined the virus genealogy.