Communication delays between the master robot and slave robot could destabilize a bilateral control system. Recent researches have shown that an adaptive controller that dynamically determines the controller gain of the master depending on the stiffness of the contact object is effective in improving stability. However, the resulting delay in estimating the stiffness of the object degrades the performance and stability of the control system. This paper therefore proposes a novel adaptive controller that utilizes the position of the contact object. The validity of the proposed controller is demonstrated through frequency domain analysis as well as simulation and experimental results. These results indicate that the proposed controller suppresses overshoot, because the proposed controller changes the controller gain before the slave robot comes into contact with an object. In addition, the control system works well if the contact object moves during a contact motion.