As a special case of perceptual hashing algorithm, subject-sensitive hashing can realize "subject-biased" integrity authentication of high resolution remote sensing (HRRS) images, which overcomes the deficiencies of existing integrity authentication technologies. However, the existing deep neural network for subject-sensitive hashing have disadvantages such as high model complexity and low computational efficiency. In this paper, we propose an efficient and lightweight deep neural network named Semi-U-net to achieve efficient subject-sensitive hashing. The proposed Semi-U-net realizes the lightweight of the network from three aspects: First, considering the general process of perceptual hashing, it adopts a semi-u-shaped structure, which simplify the model structure and prevent the model from extracting too much redundant information to enhance the robustness of the algorithm; Second, the number of model parameters and the computational cost are significantly reduced by using deep separable convolution in the entire asymmetric network; Third, the number of model parameters is further compressed by using the dropout layer several times. The experimental results show that the size of our Semi-U-Net model is only 5.38M, which is only 1/27 of MUM-net and 1/15 of MultiResUnet. The speed of the Semi-U-Net based subject-sensitive hashing algorithm is 88.6 FPS, which is 2.89 times faster than MultiResUnet based algorithm and 2.1 times faster than MUM-net Based Algorithm.