Due to its unique sensitivity to hydrogen protons, magnetic resonance sounding (MRS) is the only geophysical method that directly detects water and can provide nondestructive information on subsurface aquifer properties. The relationship between the surface MRS signal and the location and characteristics of aquifers using large-coil (typically 50–150 m) sensors has been discussed based on forward modelling and experiments. However, few researchers have studied underground MRS using a small-coil sensor. In this paper, a parametric study and a large-scale physical model test were conducted to shed light on the critical response characteristics of underground MRS using a small-coil sensor. The effects of the size and number of turns of the transmitter coil and receiver coil, the geomagnetic declination, the geomagnetic inclination, and the position, thickness, and water content of a water-bearing structure on the performance of the underground MRS were studied based on numerical simulations. Furthermore, we derived the kernel function and underground MRS signal curves for a water-bearing structure model based on the simulations. Finally, a large-scale physical model test on underground MRS using a small-coil sensor was performed using a physical test system for geological prediction of tunnels at Shandong University. The results show that the inversion results of the physical model test were in good agreement with the physical prototype results. Using both numerical modeling and physical model tests, this study showed that underground MRS using a small-coil sensor can be used to predict water-bearing structures in underground engineering.