Due to the fact that coverless information hiding can effectively resist the detection of steganalysis tools, it has attracted more attention in the field of information hiding. At present, most coverless information hiding schemes select text and image as transmission carriers, while there are few studies on emerging popular media such as video, which has more abundant contents. Taking the natural video as the carrier is more secure and can avoid the attention of attackers. In this paper, we propose a coverless video steganography algorithm based on semantic segmentation. Specifically, to establish the mapping relationship between secret information and video files effectively, this paper introduces the deep learning based on semantic segmentation network to calculate the statistical histogram of semantic information. To quickly index the sender's secret message to the corresponding video frame, we build a three-digit index structure. The receiver can extract the valid video frame from the three-digit index information and restore the secret information. On the one hand, the neural network is trained through the original image and the noisy image in this scheme; therefore, it can not only effectively resist the interference of noises, but also accurately extract the robust deep features of the image. The frames of video generate the robust mapping to the secret information after the semantic information statistics. On the other hand, semantic segmentation belongs to pixel-level segmentation, which has high requirements for network parameters, so it is difficult for attackers to decrypt and recover secret information. Since this scheme does not modify the primitiveness of video data, it can effectively resist steganalysis tools. The experimental results and analysis show that the video coverless information hiding scheme has a large capacity and a certain resistance to noise attack.