Abstract. The nonlinear harmonics within the ion motion are the fingerprint of the nonlinear fields. They are exclusively introduced by these nonlinear fields and are responsible to some specific nonlinear effects such as nonlinear resonance effect. In this article, the ion motion in the quadrupole field with a weak superimposed octopole component, described by the nonlinear Mathieu equation (NME), was studied by using the analytical harmonic balance (HB) method. Good accuracy of the HB method, which was comparable with that of the numerical fourth-order Runge-Kutta (4th RK), was achieved in the entire first stability region, except for the points at the stability boundary (i.e., β = 1) and at the nonlinear resonance condition (i.e., β = 0.5). Using the HB method, the nonlinear 3β harmonic series introduced by the octopole component and the resultant nonlinear resonance effect were characterized. At nonlinear resonance, obvious resonant peaks were observed in the nonlinear 3β series of ion motion, but were not found in the natural harmonics. In addition, both resonant excitation and absorption peaks could be observed, simultaneously. These are two unique features of the nonlinear resonance, distinguishing it from the normal resonance. Finally, an approximation equation was given to describe the corresponding working parameter, q nr , at nonlinear resonance. This equation can help avoid the sensitivity degradation due to the operation of ion traps at the nonlinear resonance condition.