Pedestrian detection and suspicious activity recognition are notable challenges in vision-based surveillance systems. However, the accuracy of pedestrian detection is influenced by a wide range of factors, including human presence, trajectory, posture, complex background, and object deformation. In this paper, we developed a pedestrian dataset that encompasses student behavior in an institution, such as cheating, stealing lab devices, dispute, and threatening scenarios. It provides uniform and steady identification annotations of pedestrians, making it suitable for pedestrian detection, tracking, and behavior detection. In addition, we also improved detection accuracy using the enhanced YOLOv5 architecture. Next, we proposed an efficient method for detection of global and local unusual behaviors. The proposed method extracts motion features that accurately describe pedestrian motion, speed, and direction, etc. The proposed method is validated on a proposed student behavior database and three additional publicly available benchmark datasets. It gives state-of-the-art detection accuracy of 96.12% and miss rate of 6.68% compared to the existing techniques. The experimental results demonstrate a substantial performance improvement in anomalous activity recognition. The final section summarizes and discusses future research directions.