This paper is based on the Chinese sign language video library, and discusses the algorithm design of video classification based on handshape recognition of key frames in video. Video classification in sign language video library is an important part of sign language arrangement and is also the premise of video feature retrieval. At present, sign language video’s handshape classification work is done manually. The accuracy and correctness of the results are quite erroneous and erroneous. In this paper, from the angle of computer image analysis, the definition and extraction of key frames are carried out, and then the region of interest is identified. Finally, an improved SURF algorithm is used to match the area of interest and the existing hand image, and the classification of the video is completed. The entire process is based on the actual development environment, and it can be used for reference based on the classification of video image features.