Digital videos are now low-cost, easy to capture and easy to share on social media due to the common feature of video recording in smart phones and digital devices. However, with the advancement of video editing tools, videos can be tampered (forged) easily for propaganda or to gain illegal advantages—ultimately, the authenticity of videos shared on social media cannot be taken for granted. Over the years, significant research has been devoted to developing new techniques for detecting different types of video tampering. In this paper, we offer a detailed review of existing passive video tampering detection techniques in a systematic way. The answers to research questions prepared for this study are also elaborated. The state-of-the-art research work is analyzed extensively, highlighting the pros and cons and commonly used datasets. Limitations of existing video forensic algorithms are discussed, and we conclude with research challenges and future directions.