Abstract-The disparity estimation problem is commonly solved using graph cut (GC) methods, in which the disparity assignment problem is transformed to one of minimizing global energy function. Although such an approach yields an accurate disparity map, the computational cost is relatively high. Accordingly, this paper proposes a hierarchical bilateral disparity structure (HBDS) algorithm in which the efficiency of the GC method is improved without any loss in the disparity estimation performance by dividing all the disparity levels within the stereo image hierarchically into a series of bilateral disparity structures of increasing fineness. To address the well-known foreground fattening effect, a disparity refinement process is proposed comprising a fattening foreground region detection procedure followed by a disparity recovery process. The efficiency and accuracy of the HBDS-based GC algorithm are compared with those of the conventional GC method using benchmark stereo images selected from the Middlebury dataset. In addition, the general applicability of the proposed approach is demonstrated using several real-world stereo images.Index Terms-Disparity estimation, foreground fattening effect, global optimization, graph cut (GC), hierarchical bilateral disparity structure (HBDS), stereo matching.