The use of fractional differential equations is a key tool in modeling non-local phenomena. Often, an efficient scheme for solving a linear system involving the discretization of a fractional operator is computing inverse fractional powers of the standard discretized Laplace operator. In this work, an exponential sum approximation for such fractional powers is derived. It is accurate over all positive real numbers larger than one, and allows to efficiently approximate the action of such operators on tensors stored in a variety of low-rank formats (CP, TT, Tucker). The results are relevant from a practical and theoretical perspective, as they predict the low-rank approximability of the solutions of these linear systems in low-rank tensor formats.