This paper describes the development of a visual tracking method of an underwater vision system installed into a UVMS (underwater vehicle-manipulator system) to manipulate ocean-floor organisms such as acanthaster. First, an execution procedure of UVMS for approaching the target is shown. Next, a novel and robust target tracking method by using particle filter framework in order to achieve the procedure is proposed. We applied the proposed method to real video images, and examined its effectiveness by comparing it with other methods. The results of some experiments that evaluate the performance of the developed method are given.