Horns, also known as headgear, are a unique structure of ruminants. As ruminants are globally distributed, the study of horn formation is critical not only for increasing our understanding of natural and sexual selection but also for the breeding of polled sheep breeds to facilitate modern sheep farming. Despite this, a significant number of the underlying genetic pathways in sheep horn remain unclear. In this study, to clarify the gene expression profile of horn buds and investigate the key genes in horn bud formation, RNA-sequencing (RNA-seq) technology was utilized to investigate differential gene expression in the horn buds and adjacent forehead skin of Altay sheep fetuses. There were only 68 differentially expressed genes (DEGs) identified, consisting of 58 up-regulated genes and 10 down-regulated genes. RXFP2 was differentially up-regulated in the horn buds and had the highest significance (p-value = 7.42 × 10−14). In addition, 32 DEGs were horn-related genes identified in previous studies, such as RXFP2, FOXL2, SFRP4, SFRP2, KRT1, KRT10, WNT7B, and WNT3. Further, Gene Ontology (GO) analysis showed that the DEGs were mainly enriched with regard to growth, development, and cell differentiation. Pathway analysis revealed that the Wnt signaling pathway may be responsible for horn development. Further, through combining the protein–protein interaction networks of the DEGs, it was found that the top five hub genes, namely, ACAN, SFRP2, SFRP4, WNT3, and WNT7B, were also associated with horn development. Our results suggest that only a few key genes, including RXFP2, are involved in bud formation. This study not only validates the expression of candidate genes identified at the transcriptome level in previous studies but also provides new possible marker genes for horn development, which may promote our understanding of the genetic mechanisms of horn formation.