In this work, we investigate activity recognition using multimodal inputs from heterogeneous sensors. Activity recognition is commonly tackled from a single-modal perspective using videos. In case multiple signals are used, they come from the same homogeneous modality, e.g. in the case of color and optical flow. Here, we propose an activity network that fuses multimodal inputs coming from completely different and heterogeneous sensors. We frame such a heterogeneous fusion as a non-local operation. The observation is that in a non-local operation, only the channel dimensions need to match. In the network, heterogeneous inputs are fused, while maintaining the shapes and dimensionalities that fit each input. We outline both asymmetric fusion, where one modality serves to enforce the other, and symmetric fusion variants. To further promote research into multimodal activity recognition, we introduce GloVid, a first-person activity dataset captured with video recordings and smart glove sensor readings. Experiments on GloVid show the potential of heterogeneous non-local fusion for activity recognition, outperforming individual modalities and standard fusion techniques. CCS CONCEPTS• Computing methodologies → Activity recognition and understanding.

No abstract

S max , ratio of sensitivity between dyad and best member; S dyad , dyad slope; S max , slope of the more sensitive participant (best member) in a dyad; S min /S max , ratio of individual sensitivity similarity; WCS, weighted confidence sharing. AbstractHumans frequently perform tasks collaboratively in daily life. Collaborating with others may or may not result in higher task performance than if one were to complete the task alone (i.e., a collective benefit). A recent study on collective benefits in perceptual decision-making showed that dyad members with similar individual performances attain collective benefit. However, little is known about the physiological basis of these results. Here, we replicate this earlier work and also investigate the neurophysiological correlates of decision-making using EEG. In a two-interval forced-choice task, co-actors individually indicated presence of a target stimulus with a higher contrast and then indicated their confidence on a rating scale. Viewing the individual ratings, dyads made a joint decision. Replicating earlier work, we found a positive correlation between the similarity of individual performances and collective benefit. We analyzed event-related potentials (ERPs) in three phases (i.e., stimulus onset, response and feedback) using explorative cluster mass permutation tests. At stimulus onset, ERPs were significantly linearly related to our manipulation of contrast differences,

Humans frequently coordinate with others in daily life. A recent study on perceptual decisionmaking showed that dyad members with similar individual performances attain a higher joint performance than the better dyad member (i.e., a collective benefit). However, little is known about the physiological basis of these results. Here, we replicate this earlier work and also investigate the neurophysiological correlates of decision-making using EEG.In a two interval forced choice task, co-actors individually indicated presence of a target stimulus with a higher contrast and then indicated their confidence on a rating scale. Viewing the individual ratings, dyads made a joint decision. Replicating earlier work, we found a positive correlation between the similarity of individual performances and collective benefit.We analyzed event related potentials (ERPs) in three phases (i.e., stimulus onset, response, and feedback) using explorative cluster mass permutation tests. At stimulus onset, ERPs were significantly linearly related to our manipulation of contrast differences, validating our manipulation of task difficulty. For individual and joint responses, we found a significant centroparietal error-related positivity for correct versus incorrect responses, which suggests that accuracy is already evaluated at the response level. At feedback presentation, we found a significant late positive fronto-central potential elicited by incorrect joint responses, suggesting a stronger emotional response to negative as compared to positive feedback. In sum, these results demonstrate that response-and feedback-related components elicited by an error-monitoring system differentially integrate conflicting information exchanged during the joint decisionmaking process.