Recent years, Big Data, Cloud Computing and the advancement of the Internet of Things (IoT) played a major role in making smart city measures feasible. During this smart city, development, busy roadside activities and appropriate parking are considered as one of the major issues in the intelligent transportation system. Especially, in the city side region, the roadside activities are creating traffic misbehaviour problems which lead to various surveillance issues. So, in this study, the focus on the effective computer vision‐related roadside surveillance system is created to reduce the unwanted traffic and misbehaviour issues. Initially, road traffic images are collected with the help of the IoT device, which is processed by noise reduction techniques to eliminate the noise. After that, the vehicle object is identified in terms of geometric pattern matching algorithm as named as compound hierarchical‐deep models. Here, the geometric matching process is used to solve the uncertainty problems during the prediction of the vehicle in roadside activities. From the object detected data, roadside activities, such as vehicle position, occupancy, gap‐related decision, have been handled with the help of a fuzzy‐based decision‐making system. Furthermore, the efficiency of the system has been evaluated using respective case studies and experimental analysis.