As a structural health monitoring (SHM) system can hardly measure all the needed responses, estimating the target response from the measured responses has become an important task. Deep neural networks (NNs) have a strong nonlinear mapping ability, and they are widely used in response reconstruction works. The mapping relation among different responses is learned by a NN given a large training set. In some cases, however, especially for rare events such as earthquakes, it is difficult to obtain a large training dataset. This paper used a convolution NN to reconstruct structure response under rare events with small datasets, and the main innovations include two aspects. Firstly, we proposed a multi-end autoencoder architecture with skip connections, which compresses the parameter space, to estimate the unmeasured responses. It extracts the shared patterns in the encoder and reconstructs different types of target responses in varied branches of the decoder. Secondly, the physics-based loss function, derived from the dynamic equilibrium equation, was adopted to guide the training direction and suppress the overfitting effect. The proposed NN takes the acceleration at limited positions as input. The output is the displacement, velocity, and acceleration responses at all positions. Two numerical studies validated that the proposed framework applies to both linear and nonlinear systems. The physics-informed NN had a higher performance than the ordinary NN with small datasets, especially when the training data contained noise.