Efficient synthesis of out-of-core algorithms using a nonlinear optimization solver

Krishnan, Sandhya; Krishnamoorthy, Sriram; Baumgartner, Gerald; Lam, Clk; Ramanujam, J.; Sadayappan, P.; Choppella, Venkatesh

doi:10.1016/j.jpdc.2005.06.017

Cited by 10 publications

(8 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The idea of automatic program transformation is present in [17]. However, this work does not take into account the characteristics of the architecture as OCAS does.…”

Section: Related Workmentioning

confidence: 99%

Automatic synthesis of out-of-core algorithms

Klonatos

Nötzli

Spielmann

et al. 2013

Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

We present a system for the automatic synthesis of efficient algorithms specialized for a particular memory hierarchy and a set of storage devices. The developer provides two independent inputs: 1) an algorithm that ignores memory hierarchy and external storage aspects; and 2) a description of the target memory hierarchy, including its topology and parameters. Our system is able to automatically synthesize memory-hierarchy and storage-device-aware algorithms out of those specifications, for tasks such as joins and sorting. The framework is extensible and allows developers to quickly synthesize custom out-of-core algorithms as new storage technologies become available.

show abstract

“…The idea of automatic program transformation is present in [17]. However, this work does not take into account the characteristics of the architecture as OCAS does.…”

Section: Related Workmentioning

confidence: 99%

Automatic synthesis of out-of-core algorithms

Klonatos

Nötzli

Spielmann

et al. 2013

Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

show abstract

“…Our start-time optimizations are similar, in spirit, to the inspector-executor model used in Chaos [Saltz et al 1995]. There is an extensive body of research on optimizing computations involving dense matrices, accessed by regular memory reference patterns [Ahmed et al 2000;Kodukula et al 1997;Lim and Lam 1998;Lim et al 2001;Krishnan et al 2003;Krishnan et al 2004]. We are not aware of any work that develops integrated compile/runtime approaches for data locality optimization, computational load balancing, and minimization of disk I/O for computations accessing semistructured and unstructured data.…”

Section: Related Workmentioning

confidence: 99%

Hypergraph Partitioning for Automatic Memory Hierarchy Management

Krishnamoorthy¹,

Catalyurek²,

Nieplocha³

et al. 2006

ACM/IEEE SC 2006 Conference (SC'06)

Self Cite

View full text Add to dashboard Cite

In this paper, we present a mechanism for automatic management of the memory hierarchy, including secondary storage, in the context of a global address space parallel programming framework. The programmer specifies the parallelism and locality in the computation. The scheduling of the computation into stages, together with the movement of the associated data between secondary storage and global memory, and between global memory and local memory, is automatically managed. A novel formulation of hypergraph partitioning is used to model the optimization problem of minimizing disk I/O. Experimental evaluation of the proposed approach using a sub-computation from the quantum chemistry domain shows a reduction in the disk I/O cost by up to a factor of 11, and a reduction in turnaround time by up to 49%, as compared to alternative approaches used in state-of-the-art quantum chemistry codes.

show abstract

“…We are developing an automatic synthesis system called the Tensor Contraction Engine (TCE) [24], to generate efficient parallel programs from high level expressions, for a class of computations expressible as tensor contractions [3,6,5,7,15,16]. Often the tensors (essentially multi-dimensional arrays) are too large to fit in memory and must be diskresident.…”

Section: Motivationmentioning

confidence: 99%

Layout transformation support for the disk resident arrays framework

et al. 2006

Self Cite

View full text Add to dashboard Cite

The Global Arrays (GA) toolkit provides a sharedmemory programming model in which data locality is explicitly managed by the programmer. It inter-operates with MPI and supports a variety of language bindings. The Disk Resident Arrays (DRA) model extends the GA programming model to secondary storage. GA and DRA together provide a convenient programming model that encourages locality-aware programming by the user, while presenting a high-level abstraction. High performance depends on the appropriate distribution of the data in the disk-resident arrays. In this paper, we discuss the addition of layout transformation support to DRA. The implementation of an efficient parallel layout transformation algorithm is done on top of existing GA/DRA functions; thus GA/DRA is itself used in implementing the enhanced DRA functionality. Experimental performance data is provided that demonstrates the effectiveness of the new layout transformation functionality.

show abstract

Efficient synthesis of out-of-core algorithms using a nonlinear optimization solver

Cited by 10 publications

References 50 publications

Automatic synthesis of out-of-core algorithms

Automatic synthesis of out-of-core algorithms

Hypergraph Partitioning for Automatic Memory Hierarchy Management

Layout transformation support for the disk resident arrays framework

Contact Info

Product

Resources

About