An understanding of inertial cavitation is crucial for biological and engineering applications such as non-invasive tissue surgeries and the mitigation of potential blast injuries. However, predictive modeling of inertial cavitation in biological tissues is hindered by the difficulties of characterizing fluids and soft materials at high strain rates, and the computational cost of calibrating biologically-relevant viscoelastic models. By incorporating a reduced-order model of inertial cavitation in the inertial microcavitation rheometry (IMR) experimental technique, we present an efficient procedure to inversely characterize viscoelastic material subjected to inertial cavitation. Instead of brute-force iteration of constitutive model parameters, the present approach directly estimates the elastic and viscous moduli according to the sizedependent scaling of bubble dynamics. Through reproduction of numerical-simulated inertial cavitation kinematics and experimental characterization of benchmark materials, we demonstrate that the proposed framework can determine the complex rate-dependent properties of soft solid with a small number of numerical simulations. The availability of this procedure will broaden the applicability of IMR for localized characterization of fluids and soft biological materials at high strain rates.