The time delay of a light signal which propagates in the gravitational field of an isolated body is considered. The body can be of arbitrary but time-independent shape and inner structure and can be in uniform rotational motion, while the center of mass of the body is assumed to be at rest. The gravitational field is given in the post-Newtonian scheme and in terms of the full set of mass-multipoles and spin-multipoles of the body. The asymptotic configuration is considered, where source and observer are located at spatial infinity from the massive body. It is found that in this asymptotic limit the higher multipole terms of time delay are related to the higher multipole terms of total light deflection. Furthermore, it is shown that the gauge terms vanish in this asymptotic configuration. In case of an axisymmetric body in uniform rotational motion, the higher multipole terms of time delay can be expressed in terms of Chebyshev polynomials. This fact allows one to determine the upper limits of the time delay for higher multipoles. These upper limits represent a criterion to identify those multipoles which contribute significantly to the time delay for a given accuracy of time measurements. It is found that the first mass-multipoles with l ≤ 8 and the first spin-multipoles with l ≤ 3 are sufficient for an accuracy on the femto-second scale of accuracy in time measurements.