We study the electrocaloric (EC) effect in bulk BaTiO3 (BTO) using molecular dynamics simulations of a first principles-based effective Hamiltonian, combined with direct measurements of the adiabatic EC temperature change in BTO single crystals. We examine in particular the dependence of the EC effect on the direction of the applied electric field at all three ferroelectric transitions, and we show that the EC response is strongly anisotropic. Most strikingly, an inverse caloric effect, i.e., a temperature increase under field removal, can be observed at both ferroelectric-ferroelectric transitions for certain orientations of the applied field. Using the generalized Clausius-Clapeyron equation, we show that the inverse effect occurs exactly for those cases where the field orientation favors the higher temperature/higher entropy phase. Our simulations show that temperature changes of around 1 K can in principle be obtained at the tetragonal-orthorhombic transition close to room temperature, even for small applied fields, provided that the applied field is strong enough to drive the system across the first order transition line. Our direct EC measurements for BTO single crystals at the cubic-tetragonal and at the tetragonal-orthorhombic transitions are in good qualitative agreement with our theoretical predictions, and in particular confirm the occurrence of an inverse EC effect at the tetragonal-orthorhombic transition for electric fields applied along the [001] pseudo-cubic direction.