A method of removing hand-induced parasitic vibration is presented based on FFT for a handheld self-mixing (SM) displacement sensor measuring mechanical vibration. Parasitic vibration has been measured by and large with solid-state accelerometers (SSAs) and, in doing so, removed. Such methods increase the complexity of the device and the cost since they need additional hardware and a complicated algorithm for elimination of parasitic vibration. The proposed method can be applied to all cases where the frequency of a measurement object is different by only a few hertz from the main frequency of the parasitic vibration as well as the case of hand-induced parasitic vibration. The error of displacement retrieval of the method was found to be less than 30 nm. This method will be a great help to the development of handheld portable SM displacement sensors.