The objective of this study was to explore the molecular mechanism for male sterility of yak hybrids based on two demethylases. Total RNA was extracted from the testes of adult yaks (n = 10) and yak hybrids (cattle–yaks, n = 10). The coding sequences (CDS) of two lysine demethylases (KDMs), KDM1A and KDM4B, were cloned by RT-PCR. The levels of KDM1A and KDM4B in yaks and cattle–yaks testes were detected using Real-time PCR and Western blotting for mRNA and protein, respectively. In addition, the histone methylation modifications of H3K36me3 and H3K27me3 were compared between testes of yaks and cattle–yaks using ELISA. The CDS of KDM1A and KDM4B were obtained from yak testes. The results showed that the CDS of KDM1A exhibited two variants: variant 1 has a CDS of 2622 bp, encoding 873 amino acids, while variant 2 has a CDS of 2562 bp, encoding 853 amino acids. The CDS of the KDM4B gene was 3351 bp in length, encoding 1116 amino acids. The mRNA and protein expression of KDM1A and KDM4B, as well as the level of H3K36me3, were dramatically decreased in the testes of cattle–yaks compared with yaks. The present results suggest that the male sterility of cattle–yaks might be associated with reduced histone methylation modifications.