The adaptive cruise control (ACC) system as a typical advanced driver assistant system (ADAS) has been commercially application in automotive industry for decades. An innovative method is proposed in this paper for scene recognition and target tracking for ACC application in some complex traffic environment. Firstly, a multi-sensor fusion method is established to estimate the curvature integrated by the quadratic programing (QP)-based lane boundaries detection, vehicle dynamics of lateral motion, and an improved Kalman filter (IKF) to introduce more measurement information into the feedback correction process. Then, the closet in-path vehicle (CIPV) can be selected according to the statistical distance between the tracked targets and the predicted driving path of ego vehicle. To distinguish the lane changing and curve driving behaviors, the trajectory models of obstacles are established as an ellipsoid domain equation and transformed into a regression model, which is recast as a standardized QP problem. Hence, the behaviors and scenes can be recognized effectively. To restrain the disturbance and improve the accuracy and robustness of target tracking, an [Formula: see text]-based switched tracking method is proposed by combining of the low pass filter (LPF) and [Formula: see text] theory. Finally, an accurate and robust tracker is provided for the CIPV by incorporating with four steps: IKF-based curvature fusion estimation, CIPV selection, QP-based scene recognition, and the [Formula: see text]-based observer. Moreover, two real car experiments are adopted and the results verify the effectiveness and real-time performance of the proposed method.