We explored the nonlinear characteristics of energy resolution (ER) for the sealed NaI (Tl) scintillator detector by using a gamma-ray spectroscopy system and Monte Carlo simulation. Our research focused on the two primary factors of energy resolution including the photomultiplier tube (PMT) voltage and the distance between the gamma-ray sources (137Cs and 60Co) and the scintillator detector. The experimental results showed that energy resolution decreased when the PMT voltage increased, and the energy resolution of NaI (Tl) detectors reached a smaller value (6.92%, 6.76%, and 6.56%), especially with the PMT voltage in the range of 575–595 V. In addition, a suitable distance between the gamma-ray source and the scintillator (5 cm) can also effectively reduce the energy resolution. We established the simulation models of the experimental NaI (Tl) detectors and simulated their energy spectra. The simulation results in the peak area agreed with the experimental results. A possible better PMT voltage choice has been proposed to obtain a smaller energy resolution.