A real time communication between deaf and hearing people is still a barrier that isolates the deaf people from the hearing world. Over ninety percent of deaf children are born to hearing parents. However, most of them can only learn how to communicate using sign language at school. One of the reasons is that the hearings parents have neither enough time nor support to learn sign language to communicate and support their children. Not surprisingly, the deaf finds difficulties in the oral-only education. Since many other hearing pupils do not even know about the existence of sign language, they cannot communicate directly with the deaf without a sign language interpreter. Therefore, to enable a face-to-face conversation between deaf and hearing people, it is important not only to sustain real time conversation between the deaf and their hearing counterparts but also to equip the hearing with basics of sign language. However, speech to sign conversion remains a challenge due to dialectal and sign language variation, speech utterance and lack of sign language written form. In this paper, a solution named Face-to-Face Conversation Deaf and Hearing people-FFCDH is proposed to address above issues. FFCDH supports real time conversation and also allows the hearing to learn the signs with the same meaning as the deaf understand. Moreover, FFCDH records the speech of the hearing and converts it into signs for the deaf. It also provides deaf with an option to adjust volume of their speech by displaying volume of their voice. The performance of the system in supporting the deaf has been evaluated by using a real test-bed. The obtained results show that English and Japanese daily conversation phrases can be recognized with over 90 percent accuracy on average. The average coherent of simple content is over 94 percent. However, when the speech includes long and complex phrases, the average accuracy and the coherent are slightly lower because the system could not comprehend long and complex context at large scope.