Efficient Management of Complex Striped Files in Active Storage

Lang

2013 IEEE International Conference on Big Data

2013

Given the growing importance of supporting dataintensive sciences and big data applications, an effective HPC I/O solution has become a key issue and has attracted intensive attention in recent years. Active storage has been shown effective in reducing data movement and network traffic as a potential new I/O solution. Existing prototypes and systems, however, are primarily designed for read-intensive applications. In addition, they generally assume that offloaded processing kernels have small computational demands, which makes this solution a poor fit for data-intensive operations that have significant computational demands, including write-intensive operations. In this research, we propose a new Multilevel Active Storage (MAS) solution. The new MAS design can support and handle both read-and writeintensive operations, as well as complex operations that have considerable computational demands. Experimental tests have been carried out and confirmed that the MAS approach is feasible and outperformed existing approaches. The new multilevel active storage design has a potential to deliver a high performance I/O solution for big data applications in HPC.

Section: B File System Level Improvementmentioning

confidence: 99%

Multilevel Active Storage for big data applications in high performance computing

Lang

2013 IEEE International Conference on Big Data

2013

2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST)

“…Recent work has focused on the application of active storage principles in the context of parallel file systems [37], [38], [17]. Piernas et al [38] proposed an approach to extend active disk techniques to the parallel file system environment in order to utilize the available computing power in the storage nodes.…”

Section: Related Workmentioning

confidence: 99%

“…Piernas et al [38] proposed an approach to extend active disk techniques to the parallel file system environment in order to utilize the available computing power in the storage nodes. In their more recent work, they showed that an active storage system can support scientific data, which is often stored in striped files and files with complex formats (e.g., netCDF [47]) [37]. These approaches are most relevant to our work in that the active storage concept is built within the context of parallel file systems, but their active storage is implemented in the user space of the Lustre file systems.…”

Section: Related Workmentioning

confidence: 99%

“…In this case, computation on the server side incurs transferring the filtered data of size N , which is only 1/N 2 of the original data volume. Accordingly, several studies have proposed active storage 1 [17], [37], [38]. Unlike the traditional storage model, an active storage model, shown in Figure 1(b), executes analysis operations on the server, thereby reducing the communication cost.…”

Section: Active Storage In Parallel File Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Enabling active storage on parallel I/O software stacks

Son

Lang

Carns

et al. 2010

Abstract-As data sizes continue to increase, the concept of active storage is well fitted for many data analysis kernels. Nevertheless, while this concept has been investigated and deployed in a number of forms, enabling it from the parallel I/O software stack has been largely unexplored. In this paper, we propose and evaluate an active storage system that allows data analysis, mining, and statistical operations to be executed from within a parallel I/O interface. In our proposed scheme, common analysis kernels are embedded in parallel file systems. We expose the semantics of these kernels to parallel file systems through an enhanced runtime interface so that execution of embedded kernels is possible on the server. In order to allow complete server-side operations without file format or layout manipulation, our scheme adjusts the file I/O buffer to the computational unit boundary on the fly. Our scheme also uses server-side collective communication primitives for reduction and aggregation using interserver communication. We have implemented a prototype of our active storage system and demonstrate its benefits using four data analysis benchmarks. Our experimental results show that our proposed system improves the overall performance of all four benchmarks by 50.9% on average and that the compute-intensive portion of the k-means clustering kernel can be improved by 58.4% through GPU offloading when executed with a larger computational load. We also show that our scheme consistently outperforms the traditional storage model with a wide variety of input dataset sizes, number of nodes, and computational loads.

2012 IEEE 31st International Performance Computing and Communications Conference (IPCCC)

“…Compared with the kernelbased implementation, the user-space approach is faster, more flexible and readily deployable. In addition, motivated by requirements of specific applications, piernas et al designed and evaluated an efficient way to manage complex striped files in active storage [17].…”

Section: Related Workmentioning

confidence: 99%

ORCA: An offloading framework for I/O-intensive applications on clusters

Zhang

Jiang

Tian

et al. 2012

Abstract-This paper presents an offloading framework -ORCA -to map I/O-intensive code to a cluster that consists of computing and storage nodes. To reduce data transmission among computing and storage nodes. our offloading framework partitions and schedules CPU-bound and I/O-bound modules to computing nodes and active storage nodes, respectively. From developer's perspective, ORCA helps them to deal with execution-path control, offloading executable code, and data sharing over a network. Powered by the offloading APIs, developers without any I/O offloading or network programming experience are allowed to write new I/O-intensive code running efficiently on clusters.We implement the ORCA framework on a cluster to quantitatively evaluate performance improvements offered by our approach. We run five real-world applications on both homogeneous and heterogeneous computing environments. Experimental results show ORCA speeds up the performance of all the five tested applications by a factor of up to 90.1% with an average of 75.5%. Moreover, the results confirm that ORCA reduces network burden imposed by I/O-intensive applications by a factor of anywhere between 35 to 68.