In this paper, we propose a method which infers an accurate disparity map for radiometrically varying stereo images. For this end, firstly, we transform the input color images to the log-chromaticity color space from which a linear relationship can be established during constructing a joint pdf between input stereo images. Based on this linear property, we present a new stereo matching cost by combining weighted mutual information and the SIFT (Scale Invariant Feature Transform) descriptor with segment-based plane-fitting constraints to robustly find correspondences for stereo image pairs which undergo radiometric variations. Experimental results show that our method outperforms previous methods and produces accurate disparity maps even for stereo images with severe radiometric differences.