A crucial problem in shape deformation analysis is to determine a deformation of a given shape into another one, which is optimal for a certain cost. It has a number of applications in particular in medical imaging.In this article we provide a new general approach to shape deformation analysis, within the framework of optimal control theory, in which a deformation is represented as the flow of diffeomorphisms generated by time-dependent vector fields. Using reproducing kernel Hilbert spaces of vector fields, the general shape deformation analysis problem is specified as an infinite-dimensional optimal control problem with state and control constraints. In this problem, the states are diffeomorphisms and the controls are vector fields, both of them being subject to some constraints. The functional to be minimized is the sum of a first term defined as geometric norm of the control (kinetic energy of the deformation) and of a data attachment term providing a geometric distance to the target shape.This point of view has several advantages. First, it allows one to model general constrained shape analysis problems, which opens new issues in this field. Second, using an extension of the Pontryagin maximum principle, one can characterize the optimal solutions of the shape deformation problem in a very general way as the solutions of constrained geodesic equations. Finally, recasting general algorithms of optimal control into shape analysis yields new efficient numerical methods in shape deformation analysis. Overall, the optimal control point of view unifies and generalizes different theoretical and numerical approaches to shape deformation problems, and also allows us to design new approaches.The optimal control problems that result from this construction are infinite dimensional and involve some constraints, and thus are nonstandard. In this article we also provide a rigorous and complete analysis of the infinite-dimensional shape space problem with constraints and of its finite-dimensional approximations.

In this paper, we define and study strong right-invariant sub-Riemannian structures on the group of diffeomorphisms of a manifold with bounded geometry. We derive the Hamiltonian geodesic equations for such structures, and we provide examples of normal and of abnormal geodesics in that infinite-dimensional context. The momentum formulation gives a sub-Riemannian version of the Euler-Arnol'd equation. Finally, we establish some approximate and exact reachability properties for diffeomorphisms, and we give some consequences for Moser theorems.

Lagrangian particle formulations of the large deformation diffeomorphic metric mapping algorithm (LDDMM) only allow for the study of a single shape. In this paper, we introduce and discuss both a theoretical and practical setting for the simultaneous study of multiple shapes that are either stitched to one another or slide along a submanifold. The method is described within the optimal control formalism, and optimality conditions are given, together with the equations that are needed to implement augmented Lagrangian methods. Experimental results are provided for stitched and sliding surfaces.2000 Mathematics Subject Classification. 58D05 49N90 49Q10 68E10.