The recent analyses in animals show that gene gain and loss fluctuate during evolutionary time. However, these studies lack the full profile of macroevolutionary trajectories as they do not consider other eukaryotic lineages and deeper evolutionary periods that go back to the origin of cellular organisms. To give a more inclusive view on the macroevolutionary patterns of genome complexity across the tree of life, we here recovered the evolutionary dynamics of gene family gain and loss ranging from the ancestor of cellular organisms to four focal species that represent animal, plant, and fungal diversity. We show that in all considered lineages the gene family content follows a common evolutionary pattern, where the number of gene families reaches the highest value at a major evolutionary and ecological transition, and then gradually decreases towards extant organisms. This pattern suggests that the genome complexity, approximated by the number of residing gene families, does not continuously increase through evolutionary time. We conclude that simplification by gene family loss is a dominant force in Phanerozoic genomes of various lineages, probably underpinned by intense ecological specializations and functional outsourcing. These findings support current theoretical predictions on the macroevolutionary change in genome complexity.