Optical Ising machines promise to solve complex optimization problems with an optical hardware acceleration advantage. Here we study the ground state properties of a nonlinear optical Ising machine realized by spatial light modulator, Fourier optics, and second harmonic generation in a nonlinear crystal. By tuning the ratio of the light intensities at the fundamental and second-harmonic frequencies, we experimentally observe two distinct ferromagnetic-to-paramagnetic phase transitions: a second-order phase transition where the magnetization changes to zero continuously and a first-order phase transition where the magnetization drops to zero abruptly as the effective temperature increases. Our experimental results are corroborated by a numerical simulation based on the Monte Carlo Metropolis-Hastings algorithm, and the physical mechanism for the distinct phase transitions can be understood with a mean-field theory. Our results showcase the great flexibility of the nonlinear optical Ising machine, which may find important potential applications in solving combinatorial optimization problems.