This is the accepted version of the paper.This version of the publication may differ from the final published version.Permanent repository link: http://openaccess.city.ac.uk/22107/ Link to published version: http://dx.Abstract-We propose a stereo vision based obstacle detection and scene segmentation algorithm appropriate for autonomous vehicles. Our algorithm is based on an innovative extension of the Stixel world which neglects computing a depth map. Ground plane and stixel distance estimation is improved by exploiting an online learned color model. Furthermore, the stixel height estimation is leveraged by an innovative joined membership scheme based on color and disparity information. Stixels are then used as an input for the semantic scene segmentation providing scene understanding, which can be further used as a comprehensive middle level representation for high-level object detectors.