Intelligent video surveillance system are extensively used in each and every sector of business. Ranging from small shops to safety systems, surveillance has become an integral part. In these fielded systems, a variety of factors can cause camera obstructions and persistent view change. The view change may adversely affect their performance. Examples include intentional blockage, noise, frame freeze, etc. which might warrant alarms. Considering the fact that the intelligent surveillance system is with very less human intervention, it is important to efficiently classify the tampered video. Analysis of the tampered videos helps in further scene investigation. The goal of the project is to use Support Vector Machines (SVM) a machine learning technique which classifies the real-time videos based on features extracted. The features selected are histogram gradients, HSV (Hue Saturation Value) and RGB (Red Blue Green) for the color based classification and edges (edge weight and direction) for the texture based classification. Further improvements are done using a deep learning technique such as CNN. Convolution neural networks make use of large amount of training data and use tensorflow framework for classification. The system accepts video inputs in mp3 or avi format. The output is the classification of tampered videos and alarm generation. Comparison between the two methodologies is done. Support vector machines gives an accuracy of 75% and convolutional neural networks give accuracy of 93%. The system is very useful to monitor all the surveillance activities.