We study the origin of the directed flow of charged particles produced in relativistic heavy-ion collisions. Three different initial conditions, Bozėk-Wyskiel, CCNU and Shen-Alzhrani, of energy density distributions are coupled to the (3+1)-dimensional viscous hydrodynamic model CLVisc, and their effects on the development of the anisotropic medium geometry, pressure gradient and radial flow are systematically compared. By comparing to experimental data at both RHIC and LHC, we find that the directed flow provides a unique constraint on the tilt of the initial medium profile in the plane spanned by the impact parameter and space-time rapidity. Within midrapidity, the counter-clockwise tilt is shown to be a crucial source of the positive/negative force by the pressure gradient along the impact parameter (x) direction at backward/forward rapidity, which drives a negative slope of the x component of the medium flow velocity with respect to rapidity, and in the end the same feature of the charged particle directed flow.