The true level crossing in the asymmetric quantum Rabi model without any obvious symmetry can be exhibited in the energy spectrum if the qubit bias is a multiple of the cavity frequency, which should imply the existence of the hidden symmetry. In this work, within a Bogoliubov operator approach, we can readily derive the symmetry operators associated with the hidden symmetry hierarchically for arbitrary multiples. The symmetry operators for small multiples in the literature can be extremely easily reproduced in our general scheme. In addition, a general parity operator is defined through the symmetry operator, which naturally includes the well-known party operator of the symmetric model. We believe that the present approach can be straightforwardly extended to other asymmetric Rabi models to find the relevant symmetry operators.