We complete our previous derivation, at the sixth post-Newtonian (6PN) accuracy, of the localin-time dynamics of a gravitationally interacting two-body system by giving two gauge-invariant characterizations of its complementary nonlocal-in-time dynamics. On the one hand, we compute the nonlocal part of the scattering angle for hyberboliclike motions; and, on the other hand, we compute the nonlocal part of the averaged (Delaunay) Hamiltonian for ellipticlike motions. The former is computed as a large-angular-momentum expansion (given here to next-to-next-to-leading order), while the latter is given as a small-eccentricity expansion (given here to the tenth order). We note the appearance of ζ(3) in the nonlocal part of the scattering angle. The averaged Hamiltonian for ellipticlike motions then yields two more gauge-invariant observables: the energy and the periastron precession as functions of orbital frequencies. We point out the existence of a hidden simplicity in the mass-ratio dependence of the gravitational-wave energy loss of a two-body system.