The left-behind human detection and tracking system based on vision with multi-model fusion and microwave radar inside the bus

Abstract

<jats:p> Left-behind humans inside the car or bus have caused a lot of accidents, so it is essential to detect the humans in vehicle. Current human detection methods rely on wearable devices, oxygen sensors, and special seat designs in vehicles, but those sensors cannot adapt to ever-changing environments. To solve those problems and especially to improve passengers’ safety on the bus, we propose a method to accomplishing human detection by fusion vision and microwave radar information in various environments in vehicle. For vision information, we use different networks to extract human and human face features, and fusion of the detection results in different models to improve human detection accuracy. The human detection model is MobileNet-V2, and the human face detection model is MTCNN. A new matching schedule and tracking objects management rule based on the Kernelized Correlation Filter tracker are designed to track the human and human face detection boxes. The microwave radar information is used to detect moving objects. Finally, the fusion vision and microwave radar detection results are implemented. Experiments show that our method has improved the human detection accuracy in vehicle, and this method can be used for detection of left-behind children on the school bus. </jats:p>

In-Vehicle Monitoring by Radar: A Review

“…On the other hand, other researchers employed the reflected energy to detect occupancy. The most common usage for this approach is the left-behind children detection to save children and pets and avoid death in excessively hot or cold conditions [2], [14], [35]- [43]. Some studies have used this approach to detect a single occupied seat [62].…”

Material identification of construction machinery based on multisource sensor information fusion

“…The algorithm then uses the Frustum Association Mechanism method to correlate the centroid detected by vision with the information detected by radar to jointly infer the 3D information of the target 3 ; Sunwoo et al proposed fusing vision and LIDAR to achieve detection of lanes, pedestrian crossings and obstacles, and the algorithm effectively eliminates the problem of false detection in complex road environments 4 . Liao et al proposed the fusion of millimeter‐wave radar and vision to detect people inside a vehicle 5 . Camera fusion to achieve forward target detection while acquiring object distance, effectively improves the safety warning distance of forward vehicles in complex environments 6 ; Hu et al proposed a vehicle tracking method that fuses millimeter‐wave radar and vision information, which first performs spatial fusion of radar and vision, then generates the target interest area by combining radar and vision information, and finally uses KCF‐KF (Kernelized Correlation Filters‐Kalman Filter) algorithm is used to track the fused vehicle 7 ; Zhao et al proposed a forward vehicle detection and tracking algorithm that fuses millimeter‐wave radar and vision, firstly, a clustering algorithm is used to reject the radar clutter target, secondly, camera data is used to detect the target, and finally, a combined filtering algorithm KCF‐EKF (kernelized Correlation Filter‐Extended Kalman Filter) was used to achieve target tracking 8…”

“…4 Liao et al proposed the fusion of millimeter-wave radar and vision to detect people inside a vehicle. 5 Camera fusion to achieve forward target detection while acquiring object distance, effectively improves the safety warning distance of forward vehicles in complex environments 6 ; Hu et al proposed a vehicle tracking method that fuses millimeter-wave radar and vision information, which first performs spatial fusion of radar and vision, then generates the target interest area by combining radar and vision information, and finally uses KCF-KF (Kernelized Correlation Filters-Kalman Filter) algorithm is used to track the fused vehicle 7 ; Zhao et al proposed a forward vehicle detection and tracking algorithm that fuses millimeter-wave radar and vision, firstly, a clustering algorithm is used to reject the radar clutter target, secondly, camera data is used to detect the target, and finally, a combined filtering algorithm KCF-EKF (kernelized Correlation Filter-Extended Kalman Filter) was used to achieve target tracking. 8 Considering that in complex environments (rain and fog, dark and weak environments, etc.…”

An Innovative multi-sensor data fusion method based on thrice deeply-fusion architecture for multi-object tracking

“…34 It formulates the data association as a constrained optimization problem, where the cost function to be minimized is a combined likelihood function evaluated by the state estimator. 35,36 Due to the huge amount of multi-sensors data in the practical application, the real-time performance demands on tracking modules are raised, and the data association and fusion will become more complex in the noisy environment. 37 Hence, a reliable multi-sensor fusion method with satisfying multi-target tracking performance and computational efficiency is important for its applications, which motivates us to propose an effective data association and fusion method that can overcome these weaknesses.…”

“…34 It formulates the data association as a constrained optimization problem, where the cost function to be minimized is a combined likelihood function evaluated by the state estimator. 35,36…”