Human Activity Recognition(HAR) is an active field of research and scientific development in which various models have been proposed using different methods for identification and categorization of activities using Machine Learning. HAR has reached a remarkable milestone in the area of computer vision. Except for applications in human-computer interactions, surveillance systems and robotics, lately it has extended its applicability in the fields of healthcare, multimedia retrieval, social networking, and education as well. It aims to bring latest technologies together to develop complex assistive system with adaptive capability and learning behaviour. HAR interprets human motion using computer and machine vision technologies to identify and detect simple and complex actions in real-world. This paper presents research made for surveillance of restricted military areas. Our scope is to develop a live monitoring system for tracking the illegal activities done in the restricted area for border security, which is an issue of concern since decades. In this we have introduced a deep learning model that learns to classify human actions without having prior knowledge. The features of image or video set are extracted and detected for detected for classifying whether the activity is illegal or not. Many harmful actions can be avoided or at least have their negative effects reduced as a result of the adoption of this concept. Finally, the activity recognition rate showed a good performance as a result of these findings