This paper deals with the problem of large number of parameters and complex calculation in video abstract generation of Fully Connected Network and Convolutional Neural Network. At the same time, the training and testing of such model need a lot of time and computer resources. We came up with a deep learning network parameter compression method based on Singular Value Decomposition(SVD) and Trucker Decomposition (TD) is proposed to generate the video summaries. The experiment was compared with other methods on TVSum and SumMe dataset, and the F1 value was 55.3% in TVSum dataset and 46.8% in SumMe dataset. At the same time, the degree of test time shortening under the same data volume is taken as the evaluation basis. The experimental results show that the proposed method achieves 1.04 times of acceleration in the SVD forward calculation, and 1.29 times of acceleration in the TD forward calculation. In a conclusion, the neural network model based on low-rank decomposition can effectively save computer resources and the time consumed by running programs.