LISSA — Live Interactive Social Skill Assistance

Ali, Mohammad Rafayet; Crasta, Dev; Li, Jin; Baretto, Agustin; Pachter, Joshua; Rogge, Ronald D.; Hoque, Mohammed Ehsan

doi:10.1109/acii.2015.7344568

Cited by 24 publications

(14 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To validate the performance of the encoder on a face-toface situation, with the camera placed in a third-person perspective as shown in Figure 1, we apply the encoder to a speed dating dataset [58]. Specifically, for the recording of each participant, we obtain the encoder outputs and calculate the proportion the individual looking into the RVE.…”

Section: Expert-rated Eye Contact Validation In Face-to-face Videosmentioning

confidence: 99%

Are You Really Looking at Me? A Feature-Extraction Framework for Estimating Interpersonal Eye Gaze From Conventional Video

Tran

Sen

Haut

et al. 2022

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

Despite a revolution in the pervasiveness of video cameras in our daily lives, one of the most meaningful forms of nonverbal affective communication, interpersonal eye gaze, i.e. eye gaze relative to a conversation partner, is not available from common video. We introduce the Interpersonal-Calibrating Eye-gaze Encoder (ICE), which automatically extracts interpersonal gaze from video recordings without specialized hardware and without prior knowledge of participant locations. Leveraging the intuition that individuals spend a large portion of a conversation looking at each other enables the ICE dynamic clustering algorithm to extract interpersonal gaze. We validate ICE in both video chat using an objective metric with an infrared gaze tracker (F1=0.846, N=8), as well as in face-to-face communication with expert-rated evaluations of eye contact (r= 0.37, N=170). We then use ICE to analyze behavior in two different, yet important affective communication domains: interrogation-based deception detection, and communication skill assessment in speed dating. We find that honest witnesses break interpersonal gaze contact and look down more often than deceptive witnesses when answering questions (p=0.004, d=0.79). In predicting expert communication skill ratings in speed dating videos, we demonstrate that interpersonal gaze alone has more predictive power than facial expressions.

show abstract

Section: Expert-rated Eye Contact Validation In Face-to-face Videosmentioning

confidence: 99%

Are You Really Looking at Me? A Feature-Extraction Framework for Estimating Interpersonal Eye Gaze From Conventional Video

Tran

Sen

Haut

et al. 2022

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

“…This interface shows how many times the user received feedback (Reminder), how long the user kept the icons green (Best Streak), and how much time the user took to adjust nonverbal behavior (Response Lag). In the past, similar interface has been used for communication skills training [12,22].…”

Section: Interfacementioning

confidence: 99%

A Virtual Conversational Agent for Teens with Autism Spectrum Disorder

Ali

Razavi

Langevin

et al. 2020

Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

Self Cite

View full text Add to dashboard Cite

We present the design of an online social skills development interface for teenagers with autism spectrum disorder (ASD). The interface is intended to enable private conversation practice anywhere, anytime using a web-browser. Users converse informally with a virtual agent, receiving feedback on nonverbal cues in realtime, and summary feedback. The prototype was developed in consultation with an expert UX designer, two psychologists, and a pediatrician. Using the data from 47 individuals, feedback and dialogue generation were automated using a hidden Markov model and a schema-driven dialogue manager capable of handling multi-topic conversations. We conducted a study with nine high-functioning ASD teenagers. Through a thematic analysis of post-experiment interviews, identified several key design considerations, notably: 1) Users should be fully briefed at the outset about the purpose and limitations of the system, to avoid unrealistic expectations. 2) An interface should incorporate positive acknowledgment of behavior change. 3) Realistic appearance of a virtual agent and responsiveness are important in engaging users. 4) Conversation personalization, for instance in prompting laconic users for more input and reciprocal questions, would help the teenagers engage for longer terms and increase the system's utility. CCS CONCEPTS • Human-centered computing → Empirical studies in HCI .

show abstract

“…We used a corpus of 15 subjects interacting with the LISSA conversational agent collected in a previous WOZ study [23]. The users were all native English speakers between the ages of 18 and 25.…”

Section: Data Preparationmentioning

confidence: 99%

Investigating Linguistic and Semantic Features for Turn-Taking Prediction in Open-Domain Human-Computer Conversation

2019

View full text Add to dashboard Cite

In this paper we address the problem of turn-taking prediction in open-ended communication between humans and dialogue agents. In a non-task-oriented interaction with dialogue agents, user inputs are apt to be grammatically and lexically diverse, and at times quite lengthy, with many pauses; all of this makes it harder for the system to decide when to jump in. As a result recent turn-taking predictors designed for specific tasks or for human-human interactions will scarcely be applicable. In this paper we focus primarily on the predictive potential of linguistic features, including lexical, syntactic and semantic features, as well as timing features, whereas past work has typically placed more emphasis on prosodic features, sometimes supplemented with non-verbal behaviors such as gaze and head movements. The basis for our study is a corpus of 15 "friendly" dialogues between humans and a (Wizard-of-Oz enabled) virtual dialogue agent, annotated for pause times and types. The model of turn-taking obtained by supervised learning predicts turn-taking points with increasing accuracy using only prosodic features, only timing and speech rate features, only lexical and syntactic features, and achieves state-of-the art performance with a mixture-of-experts model combining these features along with a semantic criterion.

show abstract

LISSA — Live Interactive Social Skill Assistance

Cited by 24 publications

References 11 publications

Are You Really Looking at Me? A Feature-Extraction Framework for Estimating Interpersonal Eye Gaze From Conventional Video

Are You Really Looking at Me? A Feature-Extraction Framework for Estimating Interpersonal Eye Gaze From Conventional Video

A Virtual Conversational Agent for Teens with Autism Spectrum Disorder

Investigating Linguistic and Semantic Features for Turn-Taking Prediction in Open-Domain Human-Computer Conversation

Contact Info

Product

Resources

About