Optical and electrical properties of the InGaAsSbN layers grown by molecular beam epitaxy (MBE) on InP substrates were studied in comparison with InGaAsN layers. It was found that the optical properties of the InGaAsSbN layer characterized by photoreflectance (PR) and photoluminescence (PL) measurements are superior to those of InGaAsN layer. In addition, electrical properties characterized by Hall measurements were also improved by adding Sb atoms. Annealing studies on photoluminescence properties of the InGaAsSbN layer revealed that the PL intensity increases and the PL spectral half width decreases upon annealing up to 700 °C. These results indicate that post‐annealing is still necessary for InGaAsSbN layers to improve crystalline quality (© 2011 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)