Sheep are among the earliest domesticated livestock species, with a wide variety of breeds present today. However, it remains unclear how far back this breed diversity goes, with formal documentation only dating back a few centuries. North European short tail breeds are often assumed to be among the oldest domestic sheep populations, even thought to represent relicts of the earliest sheep expansions during the Neolithic period reaching Scandinavia less than 6000 years ago. This study sequenced the genomes (up to 11.6X) of five sheep remains from the Baltic islands of Gotland and Åland, dating from Late Neolithic (~4100 calBP) to historical times (~1600 CE). Our findings indicate that these ancient sheep already possessed the genetic characteristics of modern North European short tail breeds, suggesting a long-term continuity of this breed type in the Baltic Sea region. Despite the wide temporal spread, population genetic analyses show high levels of affinity between the ancient genomes and they exhibit higher genetic diversity compared to modern breeds, implying a loss of diversity in recent centuries associated with breed formation. Finally, we see a potential signature of an even earlier, genetically different form of sheep in Scandinavia as these samples do not represent the first sheep in Northern Europe. Our results shed light on the development of breeds in Northern Europe specifically and the development of genetic diversity in sheep breeds and their expansion from the domestication center in general.