Photon carries linear momentum and angular momentum simultaneously. Within the light-matter interaction process, exchange of linear momentum results in optical forces, whereas exchange of angular momentum leads to optical torques. Use of optical forces (light pressure or damping) have been long and wide in quantum optomechanics, however, those of optical torque and optical angular momentum are not. Here we propose a theoretical framework based on optical angular momentum and optical torques to derive the Hamiltonians of cavity orbital and spin angular momentum optomechanical systems, respectively. Moreover, based on the method, we successfully obtain the Hamiltonian of the complex angular momentum optomechanical systems consisting of micro-cavity and several torsional oscillators, whose reflection coefficients are non-unit. Our results indicate the general applicability of our theoretical framework for the Hamiltonian of angular momentum optomechanical systems and extend the research scope of quantum optomechanics.