As-cast (Fe0.83Ga0.17)100-xYx (x=0, 3, 6 and 9) alloys were prepared by non-consumable vacuum arc melting furnace under a protective argon atmosphere. The crystal structures and surface morphologies of the alloys were studied by X-ray diffraction (XRD), optical microscope (OM) and scanning electron microscopy (SEM), combined with energy dispersive spectroscopy (EDS), respectively. The surface domain structures were observed by atomic force microscopy (AFM). The magnetostriction coefficients of the alloys were measured by strain gauging method. The results showed that the as-cast Fe83Ga17 alloy was composed only of a single phase of A2 with bcc structure, whereas the ternary Fe-Ga-Y alloys contain multiphase structure, besides the A2 phase, (FeGa)17Y1.76 new phases are observed as well, and an elemental yttrium phase appeared when the yttrium content increased to x=6 and x=9. Doping with yttrium have an effect on the change of magnetic domain structure of the binary alloy. With increasing x, the magnetostriction coefficient of the (Fe0.83Ga0.17)100-xYx alloys decreased sharply. The minimum magnetostriction coefficient is reduced to 12 ppm at the magnetic field of 426kA/m when x=9.