Depending on the actual demand of maritime security, this paper analyzes the specific requirements of video encryption algorithm for maritime monitoring system. Based on the technology of Internet of things, the intelligent monitoring system of unmanned surface vessels (USV) is designed and realized, and the security technology and network technology of the Internet of things are adopted. The USV are utilized to monitor and collect information on the sea, which is critical to maritime security. Once the video data were captured by pirates and criminals during the transmission, the security of the sea will be affected awfully. The shortcomings of traditional algorithms are as follows: the encryption degree is not high, computing cost is expensive, and video data is intercepted and captured easily during the transmission process. In order to overcome the disadvantages, a novel encryption algorithm, i.e., the improved Hill encryption algorithm, is proposed to deal with the security problems of the unmanned video monitoring system in this paper. Specifically, the Hill algorithm of classical cryptography is transplanted into image encryption, using an invertible matrix as the key to realize the encryption of image matrix. The improved Hill encryption algorithm combines with the process of video compression and regulates the parameters of the encryption process according to the content of the video image and overcomes the disadvantages that exist in the traditional encryption algorithm and decreases the computation time of the inverse matrix so that the comprehensive performance of the algorithm is optimal with different image information. Experiments results validate the favorable performance of the proposed improved encryption algorithm.