In this paper, we study the global well-posedness and scattering problem in the energy space for both focusing and defocusing the Klein-Gordon-Hartree equation in the spatial dimension d 3. The main difficulties are the absence of an interaction Morawetz-type estimate and of a Lorentz invariance which enable one to control the momentum. To compensate, we utilize the strategy derived from concentration compactness ideas, which was first introduced by Kenig and Merle [15] to the scattering problem. Furthermore, employing technique from [37], we consider a virial-type identity in the direction orthogonal to the momentum vector so as to control the momentum in the defocusing case. While in the focusing case, we show that the scattering holds when the initial data (u 0 , u 1 ) is radial, and the energy E(u 0 , u 1 ) < E(W, 0) and ∇u 0 2 2 + u 0 2 2 < ∇W 2 2 + W 2 2 , where W is the ground state.