Video classification is one of the rising fields in the video analysis. A large number of videos are accessed by people's daily from television and internet. It is easy for humans to index the video from the collection of videos which contains news, cartoon, sports, comedy and drama. Among the categories, sports video plays a vital role due to their commercial demand. There is a similarity between the different sports video which makes the classification task difficult. In this study, the sports video categorization for five categories of sports like football, cricket, volleyball, tennis and basketball is presented. The sports video categorization system uses Higher Order Spectra Features (HOSF) for the feature extraction from video frames and multiclass Support Vector Machine (SVM) classifier for the classification of videos. The system gives average classification accuracy of 93.44% using HOSF and multiclass SVM classifier.