Co-located pattern recognition is the process of identifying the sequence of patterns occurring in surveillance videos. In greater part of the existing works, the detection of rare and popular events for effective co-located pattern recognition is not concentrated. Therefore, this paper presents the automatic discovery of the co-located patterns based on rare and popular events in the video. First, the video is converted to frames, and the keyframes are preprocessed. Then, the foreground and background of the frames are estimated, and the rare and popular events are grouped using Maximum-Minimum Pixel-Per-Inch Density-Based Spatial Clustering of Applications with Noise (Max-MinPPI-DBSCAN). From the grouped image, the object detection and mapping are done, and the patch is extracted from it. Next, the edges are detected and from that, for the moving objects, motion is estimated by the Kullback-Leibler Kalman Filter (KLKF). Also, for non-moving objects, the objects/persons are tracked. From the motion estimated and tracked data, time series features are extracted. Then, the optimal features are selected using the Dung Beetle State Transition Probability Optimizer (DBSTPO). Finally, the co-located pattern is classified using a Generalized Recurrent Extreme Value Neural Network (GREVNN), and the alert message is given to the authorities. Hence, the proposed model selected the features in 53239.44ms and classified the event with 99.0723% accuracy and showed better performance than existing works.