The nonlinear restoring force (NRF) generated by the collision and friction between particles and structures is the leading cause of the complex dynamic response of the granules-structures coupled vibrating system (GSCVS). Identification of NRF can provide critical information for post-event damage diagnosis and structural design of immersed structures. However, the spatial distribution and dynamic response of the particles near the structures are diverse and complex, making it difficult to describe the NRF with an accurate mathematical model. This paper proposed a data-based nonparametric method to estimate the NRF in the GSCVS. A nonparametric model of NRF that considered the additional effects of particles on both sides of the structures and consisted of system response and undetermined coefficients was developed. The observation vector of the conventional Extended Kalman Filter (EKF) was reconstructed by the sparse measurement of the strain response. The reconstructed observation vector contains three response components: translational displacement, translational acceleration, and rotational acceleration, in which the rotational acceleration response is difficult to measure in engineering applications. The proposed EKF based on observation vector reconstruction (EKF-OVR) can identify the undetermined coefficients in the nonparametric model, and then the NRF can be calculated. Numerical studies showed that EKF-OVR achieved higher accuracy and noise robustness than the conventional EKF and the data fusion based EKF. A dynamic experimental study on granules-beam coupled vibrating system (GBCVS) was conducted, and the proposed algorithm was employed to identify the NRF of the GBCVS. The effects of excitation amplitude, particle size, and immersed depth on NRF are analyzed, and it is found that higher harmonic components in the NRF led to period doubling and chaos of the beam response.