Diet is the great determinant of bacterial composition in the rumen. However, little is known about the rumen bacterial community of Tibetan sheep living in the special ecological environment of the Qinghai-Tibetan Plateau (QTP) of China. In the present study, we used high-throughput sequencing to investigate the rumen bacterial community of Tibetan sheep associated with two primary diets: alpine pasture diet (a continuation of the sheep’s natural grazing diet) and oat (Avena sativa) hay diet on the QTP. The results showed that bacterial community richness and species diversity of the oat hay diet group were significantly greater than that of the native pasture diet group (p < 0.05). Principal co-ordinate analysis and analysis of similarities revealed that the bacterial community of the oat hay diet group was distinctly different from that of the native pasture diet group (p < 0.05). Bacteroidetes and Firmicutes were the predominant microbial phyla in the rumen. The rumen of oat-hay-fed sheep had higher proportions of Proteobacteria and novel bacteria species than the rumen of native-pasture-fed sheep. Actinobacteria, an uncommon bacterial phylum, occurred only in the oat-hay-fed group. At the genus level, Komagataeibacter, Ruminococcaceae_UCG-014, and Ruminococcaceae_NK4A214 showed significantly higher relative abundance in the oat-hay-fed sheep than in the native-pasture-fed sheep (p < 0.05). This study is the first of the QTP to employ high-throughput sequencing to examine the influence of diet on the rumen microbiome of Tibetan sheep.