It is often desirable to know the controlling mechanism of the survival probability of nano- or microscale particles in small cavities such as, e.g., confined submicron particles in fiber beds of high-efficiency filter media or ions/small molecules in confined cellular structures. Here, we address this issue based on a numerical study of the escape kinetics of inertial Brownian colloidal particles from various types of cavities with single and multiple pores. We consider both the situations of strong and weak viscous damping. Our simulation results show that as long as the thermal length is larger than the cavity size, the mean exit time remains insensitive to the medium viscous damping. On further increasing damping strength, a linear relation between the escape rate and damping strength emerges gradually. This result is in sharp contrast to the energy barrier crossing dynamics where the escape rate exhibits a turnover behavior as a function of the damping strength. Moreover, in the ballistic regime, the exit rate is directly proportional to the pore width and the thermal velocity. All these attributes are insensitive to the cavity as well as the pore structures. Further, we show that the effects of pore structure variation on the escape kinetics are conspicuously different in the low damping regimes compared to the overdamped situation. Apart from direct applications in biology and nanotechnology, our simulation results can potentially be used to understand diffusion of living or artificial micro/nano-objects, such as bacteria, virus, Janus particles, etc., where memory effects play dictating roles.