The mineralized skeleton is a critical innovation that evolved early in vertebrate history. The tissues found in dermal skeletons of ancient vertebrates are similar to the dental tissues of modern vertebrates; both consist of a highly mineralized surface hard tissue, enamel or enameloid, more resilient body dentin, and basal bone. Many proteins regulating mineralization of these tissues are evolutionarily related and form the secretory calcium-binding phosphoprotein (SCPP) family. We hypothesize here the duplication histories of SCPP genes and their common ancestors, SPARC and SPARCL1. At around the same time that Paleozoic jawless vertebrates first evolved mineralized skeleton, SPARCL1 arose from SPARC by whole genome duplication. Then both before and after the split of ray-finned fish and lobe-finned fish, tandem gene duplication created two types of SCPP genes, each residing on the opposite side of SPARCL1. One type was subsequently used in surface tissue and the other in body tissue. In tetrapods, these two types of SCPP genes were separated by intrachromosomal rearrangement. While new SCPP genes arose by duplication, some old genes were eliminated from the genome. As a consequence, phenogenetic drift occurred: while mineralized skeleton is maintained by natural selection, the underlying genetic basis has changed.