This paper addresses the problem of recognizing signs generated by a person to guide a robot. The proposed method is based on video color analysis of a moving person making signs. The analysis consists of segmentation of the middle body, arm and forearm location and recognition of the arm and forearm positions. The proposed method was experimentally tested on videos with different target colors and illumination conditions. Quantitative evaluations indicate 97.76% of correct detection of the signs in 1807 frames.