The vernissage corpus: A conversational Human-Robot-Interaction dataset

Kumar, Shubham; Sheiki, Samira; Klotz, David; Wienke, Johannes; Odobez, Jean-Marc; Wrede, Sebastien; Khalidov, Vasil; Nyugen, Laurent; Wrede, Britta; Gática-Pérez, Daniel

doi:10.1109/hri.2013.6483545

Cited by 38 publications

(21 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…results were not very accurate. Since our goal is to evaluate VFOA performance under reasonable head pose estimation, the tracker output was filtered by keeping only track segments that matched the (sparse) ground truth location available in the dataset [3], and results with too large average pose errors or no sufficient tracker recall were removed. This resulted in a dataset of 13 persons, amounting to around 100 minutes of data.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…We used the Vernissage dataset [3], containing natural interactions recorded using a Wizard of Oz approach. In each recording, that lasts around 10 minutes, Nao first engages with two participants and explains them three paintings.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…We propose an Input-Output Hidden Markov Model (IO-HMM) to integrate such information in a recognition process that also leverages on a better head-pose to gaze dynamic mapping process [9]. Experiments are conducted on a recent HRI database with available ground-truth [3] featuring natural human robot interactions. Results using either head pose ground truth or pose estimated by an automatic tracking algorithm show the viability of the method.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Leveraging the robot dialog state for visual focus of attention recognition

Sheikhi

Khalidov

Klotz

et al. 2013

Proceedings of the 15th ACM on International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

The Visual Focus of Attention (what or whom a person is looking at) or VFOA is a fundamental cue in non-verbal communication and plays an important role when designing effective human-machine interaction systems. However, recognizing the VFOA of an interacting person is difficult for a robot, since due to low resolution imaging, eye gaze estimation is not possible. Rather, head pose cue is used as a substitute for gaze, but leads to ambiguities in its interpretation as VFOA indicator. In this paper, we investigate the use of the robot conversational state, which the robot is aware of, as contextual information to improve VFOA recognition from head pose. We propose a dynamic Bayesian model that accounts for the robot state (speaking status, person he addresses, reference to objects) along with a dynamic head-togaze mapping function. Experiments on a publicly available human-robot interaction dataset, where a humanoid robot plays the role of an art guide and quiz master, shows that using such conversational context is effective in improving VFOA.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Section: Experiments and Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Leveraging the robot dialog state for visual focus of attention recognition

Sheikhi

Khalidov

Klotz

et al. 2013

Proceedings of the 15th ACM on International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

show abstract

“…Set-up and scenario: We used the Vernissage dataset (D. Jayagopi et al, 2013) to conduct our experiments, whose setup can be seen in Fig 1b. It contains 10 natural interactions with a humanoid robot "Nao", realized using a Wizard of Oz approach. Each interaction involves two participants standing in front of Nao and free to walk around and look at different objects.…”

Section: Experimental Set-up and Detailsmentioning

confidence: 99%

Combining dynamic head pose–gaze mapping with the robot conversational state for attention recognition in human–robot interactions

Sheikhi

Odobez

2015

Pattern Recognition Letters

Self Cite

View full text Add to dashboard Cite

The ability to recognize the Visual Focus of Attention (VFOA, i.e. what or whom a person is looking at) of people is important for robots or conversational agents interacting with multiple people, since it plays a key role in turn-taking, engagement or intention monitoring. As eye gaze estimation is often impossible to achieve, most systems currently rely on head pose as an approximation, creating ambiguities since the same head pose can be used to look at different VFOA targets. To address this challenge, we propose a dynamic Bayesian model for the VFOA recognition from head pose, where we make two main contributions. First, taking inspiration from behavioral models describing the relationships between the body, head and gaze orientations involved in gaze shifts, we propose novel gaze models that dynamically and more accurately predict the expected head orientation used for looking in a given gaze target direction. This is a neglected aspect of previous works but essential for recognition. Secondly, we propose to exploit the robot conversational state (when he speaks, objects to which he refers) as context to set appropriate priors on candidate VFOA targets and reduce the inherent VFOA ambiguities. Experiments on a public dataset where the humanoid robot NAO plays the role of an art guide and quiz master demonstrate the benefit of the two contributions.

show abstract

“…The quiz game is one of the most interesting forms of multiparty interaction and the robot quizmaster is a good application of speech-based interaction techniques [6], [7], [8], [14], [15], [16], [17], [18], [19]. Required tasks of a quizmaster are 1) managing the progress of a quiz game and 2) livening up the players and spectators.…”

Section: Multiparty Interaction In Quiz Gamementioning

confidence: 99%

Development of a robot quizmaster with auditory functions for speech-based multiparty interaction

Nishimuta

Yoshii

Itoyama

et al. 2014

2014 IEEE/SICE International Symposium on System Integration

View full text Add to dashboard Cite

Abstract-This paper presents a robot quizmaster that has auditory functions (i.e., ears) for moderating a multiplayer quiz game. The most basic form of oral interaction in a quiz game is that a quizmaster reads aloud a question, and each player is allowed to answer it whenever the answer comes to his or her mind. A critical problem in such oral interaction is that if multiple players speak almost simultaneously for answering, it is difficult for a "human" quizmaster to recognize overlapping answers and judge the correctness of each answer. To avoid this problem, players have conventionally been required to push a button, raise a hand, or say "Yes" to just get a right to answer a question before doing it. This requirement, however, inhibits natural oral interaction. In this paper we propose a "robot" quizmaster that can identify a player who correctly answers a question first, even when multiple players utter answers almost at the same time. Since our robot uses its own microphones (ears) embedded in the head, individual players are not required to wear small pin microphones close to their mouths. To localize, separate, and recognize overlapping utterances captured by the ears, we use a robot audition software called HARK and an automatic speech recognizer called Julius. Experimental results showed the effectiveness of our approach.

show abstract

The vernissage corpus: A conversational Human-Robot-Interaction dataset

Cited by 38 publications

References 5 publications

Leveraging the robot dialog state for visual focus of attention recognition

Leveraging the robot dialog state for visual focus of attention recognition

Combining dynamic head pose–gaze mapping with the robot conversational state for attention recognition in human–robot interactions

Development of a robot quizmaster with auditory functions for speech-based multiparty interaction

Contact Info

Product

Resources

About