The primary objective of this work is to classify the videos, which comprises the extraction of video frames and generation of features vector and identification of action sequence in the video. A deep neural network has been proposed that combines Convolutional Neural Network and Long Short Term Memory models to achieve better spatial and temporal features. Video Classification is performed by extracting the features from the frames by considering the parameters viz., frame width and height, video sequence length etc. In the proposed model, the frames extracted from a video are fed into a two-layer LSTM. The outcome of the LSTM model is forwarded to the Convolution Layers with an additional Global Average Pooling (GAP) layer in place of the fully connected layer. The frames extracted from the video are sent as an input to the twolayered Convolutional 2D layer, which is followed by Batch normalization and Max Pooling. Fully connected layer is replaced by Global Average Pooling (GAP) layer followed by dense layer. Depending on the input data, more number of batch normalization and dense layers would be added in order to achieve more accuracy. UCF101 Dataset is considered to classify the videos. The results demonstrate that the LRCN methodology outperforms both the conventional CNN method and ConvLSTM in terms of prediction accuracy. The suggested approach also provides more accurate temporal and spatial stream identification. The results have shown that the probability of the action sequence is predicted to be around 82 percentage with an accuracy of 92.86 percentage.