Secret Image Sharing (SIS) plays an important role in the field of information security. At present, from the perspective of image restoration, SIS schemes are roughly divided into two categories: traditional threshold SIS schemes and Progressive SIS (PSIS) schemes. The former provides an all-ornothing recovery mode in image reconstruction phase. If one of these shadows is damaged or lost, the reconstruction will fail. In contrast, the latter can present the original image progressively during the reconstruction phase. In this paper, a PSIS scheme based on semantic segmentation is proposed, where only specific objects or sensitive areas in the image are shared. After simple processing, the remaining background images can be publicly transmitted on insecure channels of the network. The background image will be regarded as a normal picture by people without shares. Owners who have reached the share of trapdoor numbers can recover a key target in an image. As the share increases, more key targets can be recovered. This paper is the first to propose secure secret sharing of images based on semantic targets. Through this method, it is possible to perform permission control on the target that needs to be divided and shared. Since the shared object is not the entire image, it can save storage space to a certain extent. Finally, through the analysis of the experimental results and comparison with other works, it can be concluded that the proposed scheme has a higher level of security and the amount of data that needs to be transmitted is small.