Recently proposed stable teleoperation control scheme, base on time domain passivity, is modified to remove several conservatisms. During unconstrained motion and contacting with soft and deformable environments, the two-port time domain passivity approach [21] was excessively dissipating energy even though it was stable without any energy dissipation. The main reason of this conservatism is on the fact that the time domain passivity controller does not include the external energy dissipation elements at the slave manipulator. The measured interaction force between slave and environment allow the time domain passivity observer to include the amount of energy dissipation of the slave manipulator to the monitored energy. With the modified passivity observer, reference energy following idea [24] is applied to satisfy the passivity condition. The feasibility of the developed methods is proved with experiments. Improved performance is obtained in unconstrained motion and contacting with a soft environment.