Elastic and inelastic neutron-scattering studies of Li͑Ni 1−x Fe x ͒PO 4 single crystals reveal anomalous spinwave dispersions along the crystallographic direction parallel to the characteristic wave vector of the magnetic incommensurate phase. The anomalous spin-wave dispersion ͑magnetic soft mode͒ indicates the instability of the Ising-type ground state that eventually evolves into the incommensurate phase as the temperature is raised. The pure LiNiPO 4 system ͑x =0͒ undergoes a first-order magnetic phase transition from a long-range incommensurate phase to an antiferromagnetic ͑AFM͒ ground state at T N = 20.8 K. At 20% Fe concentrations, although the AFM ground state is to a large extent preserved as that of the pure system, the phase transition is second order, and the incommensurate phase is completely suppressed. Analysis of the dispersion curves using a Heisenberg spin Hamiltonian that includes interplane and in-plane nearest-and next-nearest-neighbor couplings reveals frustration due to strong competing interactions between nearest-and next-nearest-neighbor sites, consistent with the observed incommensurate structure. The Fe substitution only slightly lowers the extent of the frustration, sufficient to suppress the incommensurate phase. An energy gap in the dispersion curves gradually decreases with the increase in Fe content from ϳ2 meV for the pure system ͑x =0͒ to ϳ0.9 meV for x = 0.2.