Optimal Multistage Algorithm for Adjoint Computation

Aupy, Guillaume; Herrmann, Julien; Hovland, Paul; Robert, Yves

doi:10.1137/15m1019222

Cited by 23 publications

(31 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2.4.2 = 2, w 1 = r 1 = 0, c 2 = ∞; Disk-Revolve. This variant of the problem has received increasingly attention in the recent years with the introduction of a second level of storage of infinite capacity but with access (write and read) costs [Aupy and Herrmann 2017;Aupy et al 2016;Schanen et al 2016;Stumm and Walther 2009]. Indeed, with the increase in the sizes of the problem, the memory was not sufficient anymore to solve the problems in a reasonnable time.…”

Section: ∈ Mmentioning

confidence: 99%

“…This model can be used to represent an architecture with a two-level storage system where the processor can write in a close cheap-to-access memory and can access a far storage, such as a disk, if needed. The problem consisting of minimizing the makespan of the Adjoint Computation problem on such an architecture was defined as DiskProb ∞ ( , c 1 , w 2 , r 2 ) in our previous work [Aupy et al 2016].…”

Section: ∈ Mmentioning

confidence: 99%

“…Aupy et al [Aupy et al 2016] designed an algorithm, denoted by Disk-Revolve, to solve DiskProb ∞ ( , c 1 , w 2 , r 2 ) optimally. This algorithm uses a sub-routine, 1D-Revolve, that uses only on checkpoint slot in the second level of storage.…”

Section: ∈ Mmentioning

confidence: 99%

See 2 more Smart Citations

H-R evolve

Herrmann

Pallez

2020

ACM Trans. Math. Softw.

Self Cite

View full text Add to dashboard Cite

We study the problem of checkpointing strategies for adjoint computation on synchronous hierarchical platforms, specifically computational platforms with several levels of storage with different writing and reading costs. When reversing a large adjoint chain, choosing which data to checkpoint and where is a critical decision for the overall performance of the computation. We introduce H-Revolve, an optimal algorithm for this problem. We make it available in a public Python library along with the implementation of several state-ofthe-art algorithms for the variant of the problem with two levels of storage. We provide a detailed description of how one can use this library in an adjoint computation software in the field of automatic differentiation or backpropagation. Finally, we evaluate the performance of H-Revolve and other checkpointing heuristics though an extensive campaign of simulation.

show abstract

Section: ∈ Mmentioning

confidence: 99%

Section: ∈ Mmentioning

confidence: 99%

See 1 more Smart Citation

H-R evolve

Herrmann

Pallez

2020

ACM Trans. Math. Softw.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Therefore, the access time to read or write a checkpoint is not negligible in contrast to the assumption frequently made for the development of checkpointing approaches. There are a few contributions that extend the available checkpointing techniques to a hierarchical checkpointing (Aupy, Herrmann, Hovland, & Robert, 2016;Schanen, Marin, Anitescu, & Z, 2016;Stumm & Walther, 2009).…”

Section: Checkpointingmentioning

confidence: 99%

An introduction to algorithmic differentiation

Gebremedhin

Walther

2019

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Algorithmic differentiation (AD), also known as automatic differentiation, is a technology for accurate and efficient evaluation of derivatives of a function given as a computer model. The evaluations of such models are essential building blocks in numerous scientific computing and data analysis applications, including optimization, parameter identification, sensitivity analysis, uncertainty quantification, nonlinear equation solving, and integration of differential equations. We provide an introduction to AD and present its basic ideas and techniques, some of its most important results, the implementation paradigms it relies on, the connection it has to other domains including machine learning and parallel computing, and a few of the major open problems in the area. Topics we discuss include: forward mode and reverse mode of AD, higher‐order derivatives, operator overloading and source transformation, sparsity exploitation, checkpointing, cross‐country mode, and differentiating iterative processes. This article is categorized under: Algorithmic Development > Scalable Statistical Methods Technologies > Data Preprocessing

show abstract

“…Then, the access time to read or write a checkpoint is not negligible, in contrast to the assumption frequently made for the development of checkpointing approaches. A few researchers have extended checkpointing techniques to a hierarchical checkpointing; see, e.g., [3,6,7]. To derive a checkpointing technique that incorporates resilience, we ignore this hierarchical nature and assume throughout that the writing or reading process for a checkpoint is performed asynchronously not interfering with the adjoint computation.…”

Section: Introductionmentioning

confidence: 99%

Checkpointing Approaches for the Computation of Adjoints Covering Resilience Issues

Bockhorn¹,

Narayanan²,

Walther³

2020

2020 Proceedings of the SIAM Workshop on Combinatorial Scientific Computing

View full text Add to dashboard Cite

Adjoint methods are an efficient approach for computing gradient information. Together with the favorable temporal complexity result for the computation of adjoints, however, comes a memory requirement that is in essence proportional to the operation count of the underlying function, for example, if algorithmic differentiation is used to provide the adjoints. For this reason, several checkpointing approaches, including binomial checkpointing, have become popular. This paper analyzes an extension of checkpointing strategies to cover restarting the computation of adjoints. Such an extension is of special interest for long-running, parallel simulations executing on large-scale computing systems, since the simulations cannot complete the calculation of the adjoints within a maximal time allocation. We describe an exhaustive search to determine checkpointing strategies with minimal runtime when covering resilience, analyze their structure and show the resulting construction principle.

show abstract

Optimal Multistage Algorithm for Adjoint Computation

Cited by 23 publications

References 8 publications

H-R evolve

H-R evolve

An introduction to algorithmic differentiation

Checkpointing Approaches for the Computation of Adjoints Covering Resilience Issues

Contact Info

Product

Resources

About