MSbO compounds (M = Mg, Co, Ni, Cu, Zn) are known in the tetragonal trirutile forms, slightly distorted monoclinically with M = Cu due to the Jahn-Teller effect. In this study, using a low-temperature exchange reaction between ilmenite-type NaSbO and molten MSO-KCl (or MgCl-KCl) mixtures, these five compositions were prepared for the first time as trigonal layered rosiaite (PbSbO)-type phases. Upon heating, they irreversibly transform to the known phases via amorphous intermediates, in contrast to previously studied isostructural MnSbO, where the stable phase is structurally related to the metastable phase. The same method was found to be applicable for preparing stable rosiaite-type CdSbO. The formula volumes of the new phases show an excellent correlation with the ionic radii (except for M = Cu, for which a Jahn-Teller distortion is suspected) and are 2-3% larger than those for the known forms although all coordination numbers are the same. The crystal structure of CoSbO was refined via the Rietveld method: P3[combining macron]1m, a = 5.1318(3) Å, and c = 4.5520(3) Å. Compounds with M = Co and Ni antiferromagnetically order at 11 and 15 K, respectively, whereas the copper compound does not show long-range magnetic order down to 1.5 K. A comparison between the magnetic behavior of the metastable and stable polymorphs was carried out. FeSbO could not be prepared because of the 2Fe + Sb = 2Fe + Sb redox reaction. This electron transfer produces an additional 5s shell for Sb and results in a volume increase. A comparison of the formula volume for the stable mixture FeSbO + 0.5SbO with that extrapolated for FeSbO predicted that the trirutile-type FeSbO can be stabilized at high pressures.