No abstract

No abstract

No abstract

Supporting deaf and hard of hearing (D/HH) people to understand natural conversation is one of the important activities of social welfare. However, currently the communication support for D/HH people is not enough in Japan. Although existing communication methods, such as sign language and lipreading, are effective in one-to-one conversation, they have several disadvantages in one-to-many conversation such as meetings or conventions. In order to support D/HH people in understanding conversation, this paper proposes a multi-modal visualization application which provides many aspects of information about speech contents. Concrete examples of visualization modes include displaying subtitles by voice recognition and showing speaker's mouth to assist lip-reading.

In order to achieve intuitive and easy operations for home network system (HNS), we have previously proposed user interface with virtual agent (called HNS virtual agent user interface, HNS-VAUI). The HNS-VAUI was implemented with MMDAgent toolkit. A user can operate appliances and services interactively through dialog with a virtual agent in a screen. However, the previous prototype heavily depends on MMDAgent, which causes a tight coupling between HNS operations and agent behaviors, and poor capability of using external information.To cope with the problem, this paper proposes a serviceoriented framework that allows the HNS-VAUI to provide richer interaction. Specifically, we decompose the tightly-coupled system into two separate services: MMC Service and MSM service. The MMC service concentrates on controlling detailed behaviors of a virtual agent, whereas the MSM service defines logic of HNS operations and dialog with the agent with richer state machines. The two services are loosely coupled to enable more flexible and sophisticated dialog in the HNS-VAUI. The proposed framework is implemented in a real HNS environment. We also conduct a case study with practical service scenarios, to demonstrate effectiveness of the proposed framework.