Without requiring targets to carry any device, device-free-based tracking is playing an important role in many emerging applications such as smart homes, fitness tracking, intruder detection, etc. While promising, current device-free tracking systems based on inexpensive commercial devices perform well in the training environment, but poorly in other environments because of different multipath reflections. This paper introduces RDTrack, a system that leverages changes in Doppler shifts, which are not sensitive to multipath, to accurately track the target. Moreover, RDTrack identifies particular patterns for fine-grained motions such as turning, walking straightly, etc., which can achieve accurate tracking. For the purpose of achieving a fine-grained device-free tracking system, this paper builds a trajectory estimating model using HMM (Hidden Markov Model) to improve the matching accuracy and reduce the time complexity. We address several challenges including estimating the tag influenced time period, identifying moving path and reducing false positives due to multipath. We implement RDTrack with inexpensive commercial off-the-shelf RFID (Radio Frequency IDentification) hardware and extensively evaluate RDTrack in a lobby, staircase and library. Our results show that RDTrack is effective in tracking the moving target, with a low tracking error of 32 cm. This accuracy is robust for different environments, highlighting RDTrack’s ability to enable future essential device-free moving-based interaction with RFID devices.