Background: Rye (Secale cereale L., 2n = 2x = 14, RR), a relative of common wheat, is a large gene resource pool for wheat improvement. Accurate and convenient identification of the rye chromatin in wheat background will facilitate the transfer and utilization of elite genes derived from rye in wheat breeding. Results: In the present study, five rye cultivars including Imperial, German White, Jingzhouheimai, Baili and Guyuan were sequenced by specific-locus amplified fragment sequencing (SLAF-seq) to develop large-scale rye-specific markers. Based on SLAF-seq and bioinformatics analyses, a total of 404 universal PCR-based and a whole set of Kompetitive allele-specific PCR (KASP) markers specific for the 14 individual rye chromosome arms were developed and validated. Additionally, two KASP markers specific for 1RS and 2RL were successfully applied in the detection of 1RS translocations in a natural population and 2RL chromosome arms in wheat-rye derived progenies that conferred adult resistance to powdery mildew. Conclusion: The 404 PCR-based markers and 14 KASP markers specific for the 14 individual rye chromosome arms developed in this study can enrich the marker densities for gene mapping and accelerate the utilization of rye-derived genes in wheat improvement. Especially, the KASP markers achieved high-throughput and accurate detection of rye chromatin in wheat background, thus can be efficiently used in marker-assisted selection (MAS). Besides, the strategy of rye-specific PCR-based markers converting into KASP markers was high-efficient and low-cost, which will facilitate the tracing of alien genes, and can also be referred for other wheat relatives.