The purpose of this paper is to present a virtual passive actuator which enables the manipulator joint to behave as a desired spring-damper system. When the manipulator interacts with an unknown environment or human operator, huge contact force will be generated with a high-stiffness trajectory tracking controller: to avoid violent collisions, the virtual passive actuator is proposed to diminish the interaction force, and adjust the system stiffness to adapt to the environment. With the mathematical model of the virtual passive actuator, the system’s stable region is determined based on the Routh–Hurwitz criterion. The influence of operation time delay, filter and model uncertainty is taken into consideration. Finally, the virtual passive actuator is incorporated into a manipulator, and validity is verified through a set of experiments. Experiment results show that, with the proposed controller, virtual passive actuator joint deformation is proportional to the external torque, and the proportional ratio is close to the stiffness setting: it can perform dynamic behavior just like a spring-damper system when subject to step external torque.