Iron selenide nanoparticles with the NiAs-like crystal structure were synthesized by thermal decomposition of iron chloride and selenium powder in a high-temperature organic solvent. Depending on the time of the compound processing at 340 °C, the nanocrystals with monoclinic (M)-Fe3Se4 or hexagonal (H)-Fe7Se8 structures as well as a mixture of these two phases can be obtained. The magnetic behavior of the monoclinic and hexagonal phases is very different. The applied-field and temperature dependences of magnetization reveal a complicated transformation between ferrimagnetic (FRM) and antiferromagnetic (AFM) structures, which can be related to the spin rotation process connected with the redistribution of cation vacancies. From XRD and Mössbauer data, the 3c type superstructure of vacancy ordering was found in the hexagonal Fe7Se8. Redistribution of vacancies in Fe7Se8 from random to ordered leads to the transformation of the magnetic structure from FRM to AFM. The Mössbauer data indicate that vacancies in the monoclinic Fe3Se4 prefer to appear near the Fe(3+) ions and stimulate the magnetic transition with the rotation of the Fe(3+) magnetic moments. Unusually high coercive force Hc was found in both (H) and (M) nanocrystals with the highest ("giant") value of about 25 kOe in monoclinic Fe3Se4. This is explained by the strong surface magnetic anisotropy which is essentially larger than the core anisotropy. Such a large coercivity is rare for materials without rare earth or noble metal elements, and the Fe3Se4-based compounds can be the low-cost, nontoxic alternative materials for advanced magnets. In addition, an unusual effect of "switching" of magnetization in a field of 10 kOe was found in the Fe3Se4 nanoparticles below 280 K, which can be important for applications.