FusionFS: Toward supporting data-intensive scientific applications on extreme-scale high-performance computing systems

Abstract

Abstract-State-of-the-art, yet decades-old, architecture of high-performance computing systems has its compute and storage resources separated. It thus is limited for modern data-intensive scientific applications because every I/O needs to be transferred via the network between the compute and storage resources. In this paper we propose an architecture that hss a distributed storage layer local to the compute nodes. This layer is responsible for most of the I/O operations and saves extreme amounts of data movement between compute and storage resources. We have designed and implemented a system prototype of this architecturewhich we call the FusionFS distributed file system-to support metadata-intensive and write-intensive operations, both of which are critical to the I/O performance of scientific applications. FusionFS has been deployed and evaluated on up to 16K compute nodes of an IBM Blue Gene/P supercomputer, showing more than an order of magnitude performance improvement over other popular file systems such as GPFS, PVFS, and HDFS.

Efficient Scheduling of Scientific Workflows Using Hot Metadata in a Multisite Cloud

“…Similar hashing schemes are used by [14,15,34] with a low memory footprint, granting access to data in almost constant time. FusionFS [49] implements a distributed metadata management based on DHTs as well. Chiron itself has a version with distributed control using an in-memory distributed DBMS [40].…”

“…More recently, Zhao et al [48] proposed using both a distributed hash table (FusionFS [49]) and a centralized database (SPADE [47]) to manage the metadata. Similarly to us, their metadata model includes both file operations and provenance information.…”

A price-performance analysis of EC2, Google Compute and Rackspace cloud providers for scientific computing

“…Scientific applications are usually hardware dependant and may be compute and I/O intensive [25,37], compute and bandwidth intensive [7,27] or compute and memory intensive [2,11,39]. Based on the research studies, weights were assigned to each evaluation metric as given in Table 11.…”

The Policy Implementation and Development Trend Analysis of Regional Economic Development in China

“…And in the evaluation system, three indicators were selected to reflect the four signs of the level of regional economic integration, regional economic development gap, and the speed of regional economic development. [1] Chen Xiushan and Yang Yan (2010) believed that the evaluation criteria of regional coordination development mainly include the following four aspects: the regional comparative advantage deviation index, the regional disparity index, the basic public service equalization degree index, the market integration degree index. [5] Tan Chenglin believed that the three criteria of regional economic developments are: the regional economic relations, regional economic growth, and regional economic differences [3].…”