Presently, video surveillance is commonly employed to ensure security in public places such as traffic signals, malls, railway stations, etc. A major challenge in video surveillance is the identification of anomalies that exist in it such as crimes, thefts, and so on. Besides, the anomaly detection in pedestrian walkways has gained significant attention among the computer vision communities to enhance pedestrian safety. The recent advances of Deep Learning (DL) models have received considerable attention in different processes such as object detection, image classification, etc. In this aspect, this article designs a new Panoptic Feature Pyramid Network based Anomaly Detection and Tracking (PFPN-ADT) model for pedestrian walkways. The proposed model majorly aims to the recognition and classification of different anomalies present in the pedestrian walkway like vehicles, skaters, etc. The proposed model involves panoptic segmentation model, called Panoptic Feature Pyramid Network (PFPN) is employed for the object recognition process. For object classification, Compact Bat Algorithm (CBA) with Stacked Auto Encoder (SAE) is applied for the classification of recognized objects. For ensuring the enhanced results better anomaly detection performance of the PFPN-ADT technique, a comparison study is made using University of California San Diego (UCSD) Anomaly data and other benchmark datasets (such as Cityscapes, ADE20K, COCO), and the outcomes are compared with the Mask Recurrent Convolutional Neural Network (RCNN) and Faster Convolutional Neural Network (CNN) models. The simulation outcome demonstrated the enhanced performance of the PFPN-ADT technique over the other methods.