An Efficient Parallel-in-Time Method for Optimization with Parabolic PDEs

Götschel, Sebastian; Minion, Michael L.

doi:10.1137/19m1239313

Cited by 27 publications

(27 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where σ is a real, negative number. For the remainder of this section, we will study the ParaOpt algorithm applied to the scalar variant (20)- (24), particularly its convergence properties as a function of σ.…”

Section: Discrete Formulationmentioning

confidence: 99%

“…Let us now write the linear ParaOpt algorithm for (20)- (24) in matrix form. For the sake of simplicity, we assume that the subdivision is uniform, that is T = ∆T , where N satisfies ∆T = N δt and M = N L, see Figure 1.…”

Section: Discrete Formulationmentioning

confidence: 99%

“…A natural area where this type of parallelization could be used effectively is in PDE constrained optimization on bounded time intervals, when the constraint is a time dependent PDE. In these problems, calculating the descent direction within the optimization loop requires solving both a forward and a backward evolution problem, so one could directly apply time parallelization techniques to each of these solves [23,24,25,26]. Parareal can also be useful in one-shot methods where the preconditioning operator requires the solution of initial value problems, see e.g.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

PARAOPT: A Parareal Algorithm for Optimality Systems

Gander¹,

Kwok²,

Salomon³

2020

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

The time parallel solution of optimality systems arising in PDE constrained optimization could be achieved by simply applying any time parallel algorithm, such as Parareal, to solve the forward and backward evolution problems arising in the optimization loop. We propose here a different strategy by devising directly a new time parallel algorithm, which we call ParaOpt, for the coupled forward and backward nonlinear partial differential equations. ParaOpt is inspired by the Parareal algorithm for evolution equations, and thus is automatically a two-level method. We provide a detailed convergence analysis for the case of linear parabolic PDE constraints. We illustrate the performance of ParaOpt with numerical experiments both for linear and nonlinear optimality systems.

show abstract

Section: Discrete Formulationmentioning

confidence: 99%

Section: Discrete Formulationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PARAOPT: A Parareal Algorithm for Optimality Systems

Gander¹,

Kwok²,

Salomon³

2020

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

show abstract

“…Recently, in Bolten et al (2017Bolten et al ( , 2018, the PFASST algorithm has been cast as a multigrid method and its convergence has been studied for synthetic diffusion-dominated and advection-dominated problems. PFASST has also been used in the context of optimal control problems (Götschel and Minion, 2019).…”

Section: Introductionmentioning

confidence: 99%

Parallel-in-time multi-level integration of the shallow-water equations on the rotating sphere

Hamon

Schreiber

Minion

2020

Journal of Computational Physics

Self Cite

View full text Add to dashboard Cite

The modeling of atmospheric processes in the context of weather and climate simulations is an important and computationally expensive challenge. The temporal integration of the underlying PDEs requires a very large number of time steps, even when the terms accounting for the propagation of fast atmospheric waves are treated implicitly. Therefore, the use of parallel-in-time integration schemes to reduce the time-to-solution is of increasing interest, particularly in the numerical weather forecasting field.We present a multi-level parallel-in-time integration method combining the Parallel Full Approximation Scheme in Space and Time (PFASST) with a spatial discretization based on Spherical Harmonics (SH). The iterative algorithm computes multiple time steps concurrently by interweaving parallel highorder fine corrections and serial corrections performed on a coarsened problem. To do that, we design a methodology relying on the spectral basis of the SH to coarsen and interpolate the problem in space.The methods are evaluated on the shallow-water equations on the sphere using a set of tests commonly used in the atmospheric flow community. We assess the convergence of PFASST-SH upon refinement in time. We also investigate the impact of the coarsening strategy on the accuracy of the scheme, and specifically on its ability to capture the high-frequency modes accumulating in the solution. Finally, we study the computational cost of PFASST-SH to demonstrate that our scheme resolves the main features of the solution multiple times faster than the serial schemes.

show abstract

“…For an introduction and overview on various parallel-in-time integration schemes for unsteady differential equations we refer the reader to the review paper in[22], and more recent development such as[23,29] …”

mentioning

confidence: 99%

Layer-Parallel Training of Deep Residual Neural Networks

Günther¹,

Ruthotto²,

Schroder³

et al. 2020

SIAM Journal on Mathematics of Data Science

View full text Add to dashboard Cite

Residual neural networks (ResNets) are a promising class of deep neural networks that have shown excellent performance for a number of learning tasks, e.g., image classification and recognition. Mathematically, ResNet architectures can be interpreted as forward Euler discretizations of a nonlinear initial value problem whose time-dependent control variables represent the weights of the neural network. Hence, training a ResNet can be cast as an optimal control problem of the associated dynamical system. For similar time-dependent optimal control problems arising in engineering applications, parallel-in-time methods have shown notable improvements in scalability. This paper demonstrates the use of those techniques for efficient and effective training of ResNets. The proposed algorithms replace the classical (sequential) forward and backward propagation through the network layers by a parallel nonlinear multigrid iteration applied to the layer domain. This adds a new dimension of parallelism across layers that is attractive when training very deep networks. From this basic idea, we derive multiple layer-parallel methods. The most efficient version employs a simultaneous optimization approach where updates to the network parameters are based on inexact gradient information in order to speed up the training process. Using numerical examples from supervised classification, we demonstrate that the new approach achieves similar training performance to traditional methods, but enables layerparallelism and thus provides speedup over layer-serial methods through greater concurrency. in particular deep residual networks (ResNets) [36], have been breaking human records in various contests and are now central to technology such as image recognition [38,43,45] and natural language processing [6,15,41].The abstract goal of machine learning is to model a function f :for input-output pairs (y, c) from a certain data set Y × C. Depending on the nature of inputs and outputs, the task can be regression or classification. When outputs are available for all samples, parts of the samples, or are not available, this formulation describes supervised, semi-supervised, and unsupervised learning, respectively. The function f can be thought of as an interpolation or approximation function.In deep learning, the function f involves a DNN that aims at transforming the input data using many layers. The layers successively apply affine transformations and element-wise nonlinearities that are parametrized by the network parameters θ. The training problem consists of finding the parameters θ such that (1.1) is satisfied for data elements from a training data set, but also holds for previously unseen data from a validation data set, which has not been used during training. The former objective is commonly modeled as an expected loss and optimization techniques are used to find the parameters that minimize the loss.Despite rapid methodological developments, compute times for training state-of-the-art DNNs can still be prohibitive, measured in the orde...

show abstract

An Efficient Parallel-in-Time Method for Optimization with Parabolic PDEs

Cited by 27 publications

References 36 publications

PARAOPT: A Parareal Algorithm for Optimality Systems

PARAOPT: A Parareal Algorithm for Optimality Systems

Parallel-in-time multi-level integration of the shallow-water equations on the rotating sphere

Layer-Parallel Training of Deep Residual Neural Networks

Contact Info

Product

Resources

About