Communications between deaf-mute and a normal person have always been a challenging task. This paper describes a way to reduce barrier of communication by developing an assistive device for deaf-mute persons. The advancement in embedded systems, provides a space to design and develop a sign language translator system to assist the dumb people, there exist a number of assistant tools. The main objective is to develop a real time embedded device for physically challenged to aid their communication in effective means. The proposed work in this paper is to implement a system without handheld gloves and sensors and by capturing the gestures continously and converting them to voice and vice versa, thus making the communication simpler for deaf and dumb people by a handheld embedded device along with the hardware setup. The effectiveness of the work is verified under MATLAB environment and further in future dedicated voice output will be produced corresponding to the text and the gesture images captured.