Efficient disk-to-disk sorting

Abstract

Many applications foreseen for exascale era should process huge amount of data. However, the IO infrastructure of current supercomputing architecture cannot be generalized to deal with this amount of data due to the need for excessive data movement from storage layers to compute nodes leading to limited scalability. There has been extensive studies addressing this challenge. Decoupled Execution Paradigm (DEP) is an attractive solution due to its unique features such as available fast storage devices close to computational units and available programmable units close to file system. In this paper we study the effectiveness of DEP for a wellknown data-intensive kernel, disk-to-disk (aka out-of-core) sorting. We propose an optimized algorithm that uses almost all features of DEP pushing the performance of sorting in HPC even further compared to other existing solutions. Advantages in our algorithm are gained by exploiting programming units close to parallel file system to achieve higher IO throughput, compressing data before sending it over network or to disk, storing intermediate results of computation close to compute nodes, and fully overlapping IO with computation. We also provide an analytical model for our proposed algorithm. Our algorithm achieves 30% better performance compared to the theoretically optimal sorting algorithm running on the same testbed but not designed to exploit the DEP architecture.

New Stereo Matching Method Based on Improved BP Algorithm

“…However, the limit of the Local methods which could only find the local optimal solution. While the global methods can achieved a excellent matching accuracy by modeling the disparity map surface as a Markov random, and various of optimization methods are used to minimize the cost solutions, such as belief propagation (BP) [22,23] and graph-cut [24][25] , dynamic programming [26] , non-linear diffusion [27] , plane-fitting algorithm is adopted to guarantee matching accuracy [28] and so on.…”