Based upon the understanding of the global topologies of the singular subset, its complement, and the hyperbolic subset in the symplectic group, in this paper we study the domains of instability for hyperbolic Hamiltonian systems and define a characteristic index for such domains. This index is defined via the Maslov-type index theory for symplectic paths starting from the identity defined by C. Conley, E. Zehnder, and Y. Long, and the hyperbolic index of symplectic matrices. The old problem of the relation between the nondegenerate local minimality and the instability of hyperbolic extremal loops in the calculus of variation is also studied via this new index for the domains of instability. * Partially supported by the NNSF and MCSEC of China and the Qiu Shi Sci. Tech. Foundation. † Associate member of ICTP 462 Yiming Long and Tianquing An NoDEA Definition 1.1 For any matrix M ∈ Sp(2n), the hyperbolic index (M ) of M is defined to be the mod 2 number of the total multiplicity of negative eigenvalues of M which are strictly less than −1.We consider the following linear Hamiltonian systems with continuous periodic symmetric coefficients:ż = JB(t)z, (1.1)where B ∈ C(S 1 , L s (R 2n )), S 1 = R/Z, Z denotes the integer set, and · denotes the derivative d dt . Let H(2n) be the set of all such linear Hamiltonian systems. Denote by γ B the fundamental solution of (1.1) starting from the identity. Then γ B ∈ P(2n). The eigenvalues of γ B (1) are called the Floquet multipliers of the system (1.1).
Definition 1.2 A matrixM ∈ Sp(2n) is hyperbolic, if all the eigenvalues of M are not on the unit circle U in the complex plane C except two of them which are 1. Denote by Sp h (2n) the subset of all hyperbolic matrices in Sp
2n). Denote by P h (2n) the subset of all hyperbolic paths in P(2n). A system (1.1) in H(2n) is hyperbolic, if its fundamental solution γ B ∈ P h (2n). Denote by H h (2n) the subset of all hyperbolic systems in H(2n). In this case we also write B ∈ H h (2n).Note that this concept of the hyperbolicity comes from the study of periodic solutions of autonomous Hamiltonian systems. As we shall show below, P h (2n) and H h (2n) contain countably infinitely many path connected components which are called the domains of instability of symplectic paths and linear Hamiltonian systems respectively. Definition 1.3 Two paths γ 0 and γ 1 ∈ P h (2n) belong to the same domain of instability, if γ 0 and γ 1 can be connected by a continuous one-parameter family of paths {γ s } 0≤s≤1 in P h (2n). In this case we write γ 0 ∼ h γ 1 . Two Hamiltonian systems of the form (1.1) with B i ∈ C(S 1 , L s (R 2n )) ∩ H h (2n) for i = 0, 1 belong to the same domain of instability, if B 0 and B 1 can be connected by a continuous one-parameter family of {B s } 0≤s≤1 in C(S 1 , L s (R 2n )) ∩ H h (2n). In this case we write B 0 ∼ h B 1 .Note that for γ 0 and γ 1 ∈ P h (2n), the facts that γ 0 ∼ γ 1 on [0, 1] along δ(·, 1) in the sense of Definition 4.2 below and that δ(·, 1) being a path in Sp h (2n) together imply γ 0 ∼ h γ 1 .A powerful t...