Crustal pore pressure, which controls the activities of earthquakes and volcanoes, varies in response to rainfall. The status of pore pressure can be inferred from observed changes in seismic velocity. In this study, we investigate the response of crustal pore pressure to rainfall in southwestern Japan based on time series of seismic velocity derived from ambient noise seismic interferometry. To consider the heterogeneity of the area, rainfall and seismic velocity obtained at each location were directly compared. We used a band-pass filter to distinguish the rainfall variability from sea level and atmospheric pressure, and then calculated the cross-correlation between rainfall and variations in S-wave velocity (Vs). A mostly negative correlation between rainfall and Vs changes indicates groundwater recharge by rainfall, which increases pore pressure. The correlations differ between locations, where most of the observation stations with clear negative cross-correlations were located in areas of granite. On the other hand, we could not observe clear correlations in steep mountain areas, possibly because water flows through river without percolation. This finding suggests that geographical features contribute to the imprint of rainfall on deep formation pore pressure. We further modelled pore pressure change due to rainfall based on diffusion mechanism. A strong negative correlation between pore pressure estimated from rainfall and Vs indicates that the Vs variations are triggered by pore pressure diffusion in the deep formation. Our modelling results show a spatial variation of diffusion parameter which controls the pore pressure in deep formation. By linking the variations in seismic velocity and crustal pore pressure spatially, this study shows that seismic monitoring may be useful in evaluating earthquake triggering processes or volcanic activity.