The Goos-Hänchen (GH) shift and the Imbert-Fedorov (IF) shift are optical phenomena which describe the longitudinal and transverse lateral shifts at the reflection interface, respectively. Here, we report the GH and IF shifts in Weyl semimetals (WSMs)-a promising material harboring low energy Weyl fermions, a massless fermionic cousin of photons. Our results show that GH shift in WSMs is valley-independent which is analogous to that discovered in a 2D relativistic materialgraphene. However, the IF shift has never been explored in non-optical systems, and here we show that it is valley-dependent. Furthermore, we find that the IF shift actually originates from the topological effect of the system. Experimentally, the topological IF shift can be utilized to characterize the Weyl semimetals, design valleytronic devices of high efficiency, and measure the Berry curvature.