Bilateral control is one of the control methods of teleoperation system. The presence of network delays between a master robot and a slave robot makes the design of bilateral control system challengeable. Because, it seriously deteriorates the performance and possibly makes the system unstable. The communication disturbance observer (CDOB) has been proposed to compensate time delay effect without a time delay model. The effectiveness of CDOB with acceleration based four-channel bilateral control system has been confirmed experimentally so far. The acceleration based four-channel bilateral control system utilizes the information of the master/slave position and force. Position and force control are realized in two decomposed virtual modal spaces, i.e. differential mode and common mode. In the acceleration based four-channel bilateral control system with CDOB, position control is stable, but force control is not stable. Thus, the control system is stable during free motion, but the system can be destabilized when the slave robot contacts with the environment. In this paper, a structural approach is applied to improve the stability of force control of bilateral control system. The proposed structure is derived from modal space analysis in differential mode and common mode. By using the proposed structure, position control and force control of the bilateral control system are stabilized. The validity is confirmed by experimental results in the case of constant delay and time varying delay.