In this paper, we prove several new Hardy type inequalities (such as the weighted Hardy inequality, weighted Rellich inequality, critical Hardy inequality and critical Rellich inequality) for radial derivations (i.e., the derivation along the geodesic curve) on Cartan-Hadamard manifolds. By Gauss lemma, our new Hardy inequality are stronger than the classical one. We also established the improvements of these inequalities in terms of sectional curvature of underlying manifolds which illustrate the effect of curvature to these inequalities. Furthermore, we obtain some improvements of Hardy and Rellich inequality in hyperbolic space H n . Especially, we show that our new Rellich inequality is indeed stronger the classical one in hyperbolic space H n . for any n ≥ 2 (see, e.g., [14]). The constant (n − 1) n /n n is the best constant in (1.2). It also was shown in [14] that (1.2) is equivalent to the critical case of the Sobolev-Lorentz inequality. It is remarked that (1.2) is not invariant under the scaling as (1.1). A scaling invariant version of (1.2) (nowaday called the critical Hardy inequality) was recently established by Ioku and Ishiwata [25],Again, the constant (n − 1) n /n n is sharp. The inequality (1.3) in bounded domains was also discussed in [25]. It is surprise that the critical Hardy inequality (1.3) is equivalent to the Hardy inequality (1.1) in larger dimension spaces (see [45]). We refer the readers to [26] for a global scaling invariant version of (1.3) which we do not mention here.Recently, there is an enourmous work to generalize the Hardy inequality to many different settings. For examples, the fractional Hardy inequality was established in [17-19, 38, 47] and references therein. The Hardy inequality also was proved on group structure, e.g., on Heisenberg groups [8,10,21,40], on polarisable groups [22], on Carnot groups [28,33], on stratified groups [7,41], and on more general homogeneous groups [42][43][44]. The Hardy inequality on Riemannian manifold (M, g) was studied by Carron [6] in the weighted L 2 −form under some geometric assumption on the weighted function ρ. More precisely, he proved the following inequalitywhere ρ is a nonnegative function on M such that |∇ g ρ| g = 1, ∆ g ρ ≥ γ/ρ, and dV g , ∇ g , ∆ g and |·| g denote the volume element, gradient, Laplace-Beltrami operator and the length of a vector field with respect to the Riemannian metric g on M respectively, and the set ρ −1 (0) is a compact set of zero capacity. Under the same hypotheses on the function ρ, Kombe andÖzaydin [29] extended the result of Carron to the case p = 2 and they presented an application to the hyperbolic space H n with ρ being the distance function from the origin point 0. We refer the reader to [9] and the references therein for more results in this direction. The sharp Hardy inequality on Cartan-Hadamard manifolds (M, g) was recently obtained by Yang, Su and Kong [48]