Vision‐based driver assistance systems (VB‐DAS) still assume a driver in the ego‐vehicle (i.e., that vehicle where the system operates in); autonomous driving can benefit from solutions designed for VB‐DAS. The article reviews research subjects for VB‐DAS and provides an extensive bibliography. It starts with defining basic concepts and notations in the field, also highlighting the importance of performance evaluation for VB‐DAS solutions. The article then reviews briefly safety and comfort functionalities (avoidance of blind spots, night vision, and virtual windshield), before discussing in detail solutions for basic and midlevel environment perception. This involves distance and motion computation, ego‐motion understanding, obstacle detection, tasks of object tracking (vehicles or pedestrians), and the detection and recognition of infrastructure key elements (road, lane, traffic signs, and free space). Representative VB‐DAS examples (e.g., driver monitoring and driver‐environment understanding, speed adaptation, queuing, parking, blind spot supervision, lane departure warning, wrong lane detection, intelligent headlamp control, or intercar communication) are also described. Outlook toward autonomous driving and road‐environment modeling conclude the article.