Carbon cycling in anoxic marine sediments is dependent on uncultured microbial communities. Niches of heterotrophic microorganisms are defined by organic matter (OM) type and the different phases in OM degradation. We investigated how OM type defines microbial communities originating from organic-rich, anoxic sediments from the Baltic Sea. We compared changes in the sediment microbial community, after incubation with different stable isotope labeled OM types [i.e., particulate algal organic matter (PAOM), protein, and acetate], by using DNA stable isotope probing (DNA-SIP). Incorporation of 13C and/or 15N label was predominantly detected in members of the phyla Planctomycetes and Chloroflexi, which also formed the majority (>50%) of the original sediment community. While these phylum-level lineages incorporated label from all OM types, phylogenetic analyses revealed a niche separation at the order level. Members of the MSBL9 (Planctomycetes), the Anaerolineales (Chloroflexi), and the class Bathyarchaeota, were identified as initial degraders of carbohydrate-rich OM, while other uncultured orders, like the CCM11a and Phycisphaerales (Planctomycetes), Dehalococcoidia, and JG30-KF-CM66 (Chloroflexi), incorporated label also from protein and acetate. Our study highlights the importance of initial fermentation of complex carbon pools in shaping anoxic sediment microbial communities and reveals niche specialization at the order level for the most important initial degraders in anoxic sediments.