Video surveillance systems obtain a great interest as application-oriented studies that have been growing rapidly in the past decade. The most recent studies attempt to integrate computer vision, image processing, and artificial intelligence capabilities into video surveillance applications. Although there are so many achievements in the acquisition of datasets, methods, and frameworks published, there are not many papers that can provide a comprehensive picture of the current state of video surveillance system research. This paper provides a comprehensive and systematic review on the literature from various video surveillance system studies published from 2010 through 2019. Within a selected study extraction process, 220 journal-based publications were identified and analyzed to illustrate the research trends, datasets, methods, and frameworks used in the field of video surveillance, to provide an in-depth explanation about research trends that many topics raised by researchers as a focus in their researches, to provide references on public datasets that are often used by researchers as a comparison and a means of developing a test method, and to give accounts on the improvement and integration of network infrastructure design to meet the demand for multimedia data. In the end of this paper, several opportunities and challenges related to researches in the video surveillance system are mentioned.INDEX TERMS Artificial intelligence, cloud video surveillance, intelligent video surveillance, video surveillance framework.This study is conducted as follows: the methodology of the study is presented in Section 2. The outcomes and answers to the research questions are then discussed in Section 3. Finally, the study is summarized in the last section.