Optically pumped alkali atomic magnetometers based on measuring the Zeeman shifts of the atomic energy levels are widely used in many applications because of their low noise and cryogen-free operation. When alkali atomic magnetometers are operated in an unshielded geomagnetic environment, the nonlinear Zeeman effect may become non-negligible at high latitude and the Zeeman shifts are thus not linear to the strength of the magnetic field. The nonlinear Zeeman effect causes broadening and partial splitting of the magnetic resonant levels, and thus degrades the sensitivity of the alkali atomic magnetometers and causes heading error. In this work, we find that the nonlinear Zeeman effect also influences the frequency response of the alkali atomic magnetometer. We develop a model to quantitatively depict the frequency response of the alkali atomic magnetometer when the nonlinear Zeeman effect is non-negligible and verify the results experimentally in an amplitude-modulated Bell–Bloom cesium magnetometer. The proposed model provides general guidance on analyzing the frequency response of the alkali atomic magnetometer operating in the Earth’s magnetic field. Full and precise knowledge of the frequency response of the atomic magnetometer is important for the optimization of feedback control systems such as the closed-loop magnetometers and the active magnetic field stabilization with magnetometers. This work is thus important for the application of alkali atomic magnetometers in an unshielded geomagnetic environment.