Overview of the MPI-IO Parallel I/O Interface

This paper considers key ideas in the design of out-of-core dense LU factorization routines. A left-looking variant o f t h e LU factorization algorithm is shown to require less I O to disk than the rightlooking variant, and is used to develop a parallel, out-of-core implementation. This implementation makes use of a small library of parallel I O routines, together with ScaLAPACK and PBLAS routines. Results for runs on an Intel Paragon are presented and interpreted using a simple performance model.

show abstract

“…Moreover, disk storage is cheap and easy to install. Consider the system requirements if we w ant to factor a 10 5 …”

Section: Discussionmentioning

confidence: 99%

Key concepts for parallel out-of-core LU factorization

Dongarra¹,

Hammarling²,

Walker³

1997

Parallel Computing

View full text Add to dashboard Cite

show abstract

“…Through this API, the programs can interact with the database tables without getting involved with low-level SQL-like commands. Our MDMS library is built on top of MPI-I/O [9], the emerging parallel I/O standard. MPI-I/O provides many I/O optimization methods such as collective I/O, data sieving, asynchronous I/O, and so forth.…”

Section: Mdms Apimentioning

confidence: 99%

“…These techniques can be ~lassified into three categories: the parallel file system and ran-time system optimizations [21,7,9,18,20,15], compiler optimizations [4,19,16], and application analysis and optimization [19,6,28,16]. Brown et al [5] proposed a meta-data system on top of HPSS using DB2 DBMS.…”

Section: Related Workmentioning

confidence: 99%

“…Third, the file system policies and optimization parameters are in general hard-coded within the file system and, consequently, work for only a small set of access patterns. While mntime systems and libraries like MPI-IO [9,33] and others [35,3,7] present users with higher-level, more structured interfaces, the excessive number of calls to select from, each with several parameters, make the user's job very difficult. Also, the usability of these libraries depends largely on how well user's access patterns and library calls' functionality match [20].…”

mentioning

confidence: 99%

See 1 more Smart Citation

A novel application development environment for large-scale scientific computations

Shen

Liao

Choudhary

et al. 2000

Proceedings of the 14th International Conference on Supercomputing

View full text Add to dashboard Cite

d h a r y , G. M e m i k , M . Kandemir,* S. M o r e , G. T h i r u v a t h u k a l , t and A, S i n g h t C e n t e r for Parallel a n d D i s t r i b u t e d C o m p u t i n g D e p a r t m e n t o f E l e c t r i c a l and C o m p u t e r E n g i n e e r i n g N o r t h w e s t e r n U n i v e r s i t y E v a n s t o n , I L 60208 { x h s h e n , w k l i a o , c h o u d h a r , m e m i k , s s m o r e } @ e c e . n w u . e d u AbstractEffective high-level data management is becoming an important issue with more and more scientific applications manipulating huge amounts of secondary-storage and tertiary-storage data using parallel processors. A major problem facing the current solutions to this data management problem is that these solutions either require a deep understanding of specific data storage architectures and file layouts to obtain the best performance. In this paper, we discuss the design, implementation, and evaluation of a novel application development environment for scientific computations. This environment includes a number of components that make it easy for the programmers to code and run their applications without much programming effort, and at the same time, to harness the available computational and storage power on parallel architectures. Embarking on this ambitious goal, we first present a performanceoriented meta-data management system that governs data flow between storage devices and applications. Another component of our environment is a data analysis and visualization tool which has been integrated with the recta-data management system, storage subsystem, and user applications. We also present an automatic code generator component (ACG) to help users utilize the information in the meta-data management system when they are developing new applications. All these components are tied together using an integrated Java graphical user interface (IJ-GUI) through which the user can launch her applications, can query the meta-data management system to obtain accurate information about the datasets she is interested in and about the current state of the storage devices, and can carry out data analysis and visualization, all in a unified framework. Finally, we present performance numbers from our initial implementation. Our results demonstrate that our novel application development environment provides both ease-of-use and high performance for large-scale, I/O-intensive scientific applications.

show abstract

“…We note that the same model has proved successful in the GA library: operations that move data between two global memory locations are collective while transfer between global and local memory is noncollective. This collective I/O strategy has been adopted in many other projects for similar reasons [33,34,35]. A DRA asynchronous interface to its I/O operation permits applications to overlap time-consuming I/O with computation.…”

Section: Disk Resident Arraysmentioning

confidence: 99%