Traffic congestion detection based on vehicle detection and tracking algorithms is one of the key technologies for intelligent transportation systems. However, in expressway surveillance scenarios, small vehicle size and vehicle occlusion present severe challenges for this method, including low vehicle detection accuracy and low traffic congestion detection accuracy. To address these challenges, this paper proposes an improved version of the CrowdDet algorithm by introducing the Involution operator and bi-directional feature pyramid network (BiFPN) module, which is called IBCDet. The proposed IBCDet module can achieve higher vehicle detection accuracy in expressway surveillance scenarios by enabling long-distance information interaction and multi-scale feature fusion. Additionally, a vehicle-tracking algorithm based on IBCDet is designed to calculate the running speed of vehicles, and it uses the average running speed to achieve traffic congestion detection according to the Chinese expressway level of serviceability (LoS) criteria. Adequate experiments are conducted on both the self-built Nanjing Raoyue expressway monitoring video dataset (NJRY) and the public dataset UA-DETRAC. The experimental results demonstrate that the proposed IBCDet outperforms the commonly used object detection algorithms in both vehicle detection accuracy and traffic congestion detection accuracy.