Indirect identification approaches through structural responses have proven effective for wind load estimation in real-world engineering. Currently, methods for identifying wind loads mainly rely on theoretical inverse identification, with rare research based on the mapping relationship between structural responses and wind loads through machine learning. In this paper, a scheme for identifying full-field wind loads using a recursive convolutional neural network (CNN) inspired by physical mechanisms is proposed. The recursive form of the network, as well as the inspiration for its inputs and outputs, is inspired by the spatial correlation and the mapping relationship between wind loads and structural responses. Thus, the network inputs comprise a fusion of structural acceleration and inter-story displacement responses, while the network outputs represent the independent wind loads on structures. Notably, mismatch test is employed by the network, wherein the training and testing datasets originate from entirely different sources. Specifically, during training, Gaussian white noises that simulate wind loads are utilized, while real wind load data are used for testing. The generalization of the proposed scheme is demonstrated through the identification of full-field wind loads generated by different stationary or non-stationary wind spectra of the 76-story wind-excited benchmark building. Furthermore, the proposed scheme is validated by identifying the full-field wind loads of a 67-story shear wall structure with wind tunnel test data.