LAEO-Net: Revisiting People Looking at Each Other in Videos

Abstract

Figure 1: Intimacy or hostility? Head pose, along with body pose and facial expressions, is a rich source of information for interpreting human interactions. Being able to automatically understand the non-verbal cues provided by the relative head orientations of people in a scene enables a new level of human-centric video understanding. Green and red/orange heads represent LAEO and non-LAEO cases, respectively. Video source of second row: https://youtu.be/B3eFZMvNS1UAbstract Capturing the 'mutual gaze' of people is essential for understanding and interpreting the social interactions between them. To this end, this paper addresses the problem of detecting people Looking At Each Other (LAEO) in video sequences. For this purpose, we propose LAEO-Net, a new deep CNN for determining LAEO in videos. In contrast to previous works, LAEO-Net takes spatio-temporal tracks as input and reasons about the whole track. It consists of three branches, one for each character's tracked head and one for their relative position. Moreover, we introduce two new LAEO datasets: UCO-LAEO and AVA-LAEO. A thorough experimental evaluation demonstrates the ability of LAEO-Net to successfully determine if two people are LAEO and the temporal window where it happens. Our model achieves state-of-the-art results on the existing TVHID-LAEO video dataset, significantly outperforming previous approaches. Finally, we apply LAEO-Net to social network analysis, where we automatically infer the social relationship between pairs of people based on the frequency and duration that they LAEO.

RealHePoNet: a robust single-stage ConvNet for head pose estimation in the wild

“…However, these results highlight Fig. 7: Comparison between detections found with the model in the work by Marín-Jiménez et al [28] (left) and the Viola-Jones detector used by Patacchiola et al (right) Notice the difference in precision, making the CNN-based head detector much more suitable for this task than the Viola-Jones detector (best viewed on digital format).…”

“…For our testing procedure, we have used one of the models provided by the authors in the Hopenet repository [40] (300W-LP, alpha 1, robust to image quality); this model has been chosen as it should be the most suitable to be applied over real-world pictures as the ones appearing in the AFLW dataset. The input images correspond to the portion of the AFLW dataset used to test our model; they have been obtained using the head detector in the work by Marín-Jiménez et al [28], but they have been resized to 224 × 224; also, as the ResNet50 model uses color pictures as input, the pictures have not been converted to grayscale.…”

“…Human head pose estimation is useful in many situations: for instance in vehicles (detecting if the driver of a vehicle is paying attention to the road [31]), human-computer interaction (detecting where the user's attention is being drawn [44]), social interaction understanding (detecting if people is looking at each other [28]), video surveillance systems [18,36] or to aid various aerial cinematography tasks [33].…”

Three-Dimensional Human Head Reconstruction Using Smartphone-Based Close-Range Video Photogrammetry

“…A convolutional neural network (CNN) single-shot detector (SSD) [80] is used for head detection [81] in the images (Figure 2). The model adopted in this research was developed by the authors of LAEO-Net [82]. The model's suitability for the task was evaluated by manually revising the head detection results on the collected dataset of 19 videos.…”

“…The numbers on the boxes specify the shape of the feature layers. The model adopted in this research is developed by the authors of LAEO-Net [82].…”