Quantum criticality usually occurs in many-body systems. Recently it was shown that the quantum Rabi model, which describes a two-level atom coupled to a single model cavity field, presents quantum phase transitions from a normal phase to a superradiate phase when the ratio between the frequency of the two level atom and the frequency of the cavity field extends to infinity. In this work, we study quantum phase transitions in the quantum Rabi model from the fidelity susceptibility perspective. We found that the fidelity susceptibility and the generalized adiabatic susceptibility present universal finite size scaling behaviors near the quantum critical point of the Rabi model if the ratio between frequency of the two level atom and frequency of the cavity field is finite. From the finite size scaling analysis of the fidelity susceptibility, we found that the adiabatic dimension of the fidelity susceptibility and the generalized adiabatic susceptibility of fourth order in the Rabi model are 4/3 and 2, respectively. Meanwhile the correlation length critical exponent and the dynamical critical exponent in the quantum critical point of the Rabi model are found to be 3/2 and 1/3 respectively. Since the fidelity susceptibility and the generalized adiabatic susceptibility are the moments of the quantum noise spectrum which is directly measurable by experiments in linear response regime, the scaling behavior of the fidelity susceptibility in the Rabi model could be tested experimentally. The simple structure of the quantum Rabi model paves the way for experimentally observing the universal scaling behavior of the fidelity susceptibility at a quantum phase transition. the quantum information theory and condensed matter physics and shall deepen our understanding on the various condensed matter phenomena.