The purpose of this paper is to study the exceptional eigenvalues of the asymmetric quantum Rabi models (AQRM), specifically, to determine the degeneracy of their eigenstates. Here, the Hamiltonian H ε Rabi of the AQRM is defined by adding the fluctuation term εσ x , with σ x being the Pauli matrix, to the Hamiltonian of the quantum Rabi model, breaking its Z 2 -symmetry. The spectrum of H ε Rabi contains a set of exceptional eigenvalues, considered to be remains of the eigenvalues of the uncoupled bosonic mode, which are further classified in two types: Juddian, associated with polynomial eigensolutions, and non-Juddian exceptional. We explicitly describe the constraint relations for allowing the model to have exceptional eigenvalues. By studying these relations we obtain the proof of the conjecture on constraint polynomials previously proposed by the third author. In fact, we prove that the spectrum of the AQRM possesses degeneracies if and only if the parameter ε is a half-integer. Moreover, we show that non-Juddian exceptional eigenvalues do not contribute any degeneracy and we characterize exceptional eigenvalues by representations of sl 2 . Upon these results, we draw the whole picture of the spectrum of the AQRM. Furthermore, generating functions of constraint polynomials from the viewpoint of confluent Heun equations are also discussed.