This paper presents a survey of activity recognition and understanding the behaviour of human activity in video sequence. The major goal of this paper is to provide a general review on the overall process of a surveillance system used in the current trend. Visual surveillance system is directed on automatic identification of events of interest, especially on tracking and classification of moving objects. The processing step of the video surveillance system includes the following stages: Surrounding model, object representation, object tracking, activity recognition and behaviour understanding. It describes techniques that use to define a general set of activities that are applicable to a wide range of scenes and environments in video sequence. The review methods used for real-time surveillance through a set of events for further analysis triggering, including virtual fencing, speed profiling, behaviour classification, anomaly detection, and object interaction.