For autonomously acting robots and driver assistance systems powerful optical stereo sensor systems are required. Object positions and environmental conditions have to be acquired in real-time. In this paper a hardware-software co-design is applied, acting within the presented stereophotogrammetric system. For calculation of the depth map an optimized algorithm is implemented as a hierarchical parallel hardware solution. By adapting the image resolution to the distance, real-time processing is possible. The object clustering and the tracking is realized in a processor. The density distribution of the disparity in the depth map (disparity histogram) is used for object detection. A Kalman filter stabilizes the parameters of the results.