Anomaly analysis is an important component of any surveillance system. In recent years, it has drawn the attention of the computer vision and machine learning communities. In this article, our overarching goal is thus to provide a coherent and systematic review of state-of-the-art techniques and a comprehensive review of the research works in anomaly analysis. We would like to provide a broad vision of computational models, datasets, metrics, extensive experiments, and what anomaly analysis can do in images and videos. Intensively covering nearly 200 publications we review i) anomaly related surveys, ii) taxonomy for anomaly problems, iii) the computational models, iv) the benchmark datasets for studying abnormalities in images and videos, and v) the performance of state-of-the-art methods in this research problem. In addition, we provide insightful discussions and pave way to the future work.