When human operator is in the control loop of a robotic system, which operates in unstructured, complex and dynamic environments such as those in construction and steel bridge maintenance, human factors will have significant effect on the operational performance and control of the robot. This paper aims to investigate this effect and develop efficient methods for operational performance improvement. An eXtended Hand Movement (XHM) model for eye-guided hand movement is developed with the aim of providing natural and comfortable interaction between a human operator and a robot. A haptic force-speed-accuracy control method is then designed based on the XHM model and implemented in a steel bridge maintenance robotic system. An experiment is conducted to verify the model and control method. Experimental results demonstrated significant improvement in control accuracy and operational performance in a steel bridge maintenance robotic system.