Attention in Multimodal Neural Networks for Person Re-identification

Lejbolle, Aske R.; Krogh, Benjamin; Nasrollahi, Kamal; Moeslund, Thomas B.

doi:10.1109/cvprw.2018.00055

Cited by 16 publications

(26 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In RGB-D-CNN, Lejbolle et al [29] started with a two-flow Convolutional Neural Network (CNN) (one for RGB and one for depth) and a final fusion layer. Then they improved this approach with a multimodal attention network called MAT [30], adding an attention module to extract local and discriminative features that were fused with globally extracted features. In another work, Lejbolle et al [31] presented a SLATT network with two types of attention modules (one spatial and one layer-wise).…”

Section: Resultsmentioning

confidence: 99%

Open-World Person Re-Identification With RGBD Camera in Top-View Configuration for Retail Applications

Martini¹,

Paolanti²,

Frontoni³

2020

IEEE Access

View full text Add to dashboard Cite

Person re-identification (re-ID) is currently a notably topic in the computer vision and pattern recognition communities. However, most of the existing works on re-ID have been designed for closed world scenarios, rather than more realistic open world scenarios, limiting the practical application of these re-ID techniques. In a common real-world application, a watch-list of known people is given as the gallery/target set for searching through a large volume of videos where the people on the watch-list are likely to return. This aspect is fundamental in retail for understanding how customers schedule their shopping. The identification of regular and occasional customers allows to define temporal purchasing profiles, which can put in correlation the customers' temporal habits with other information such as the amount of expenditure and number of purchased items. This paper presents the first attempt to solve a more realistic re-ID setting, designed to face these important issues called Top-View Open-World (TVOW) person re-id. The approach is based on a pretrained Deep Convolutional neural Network (DCNN), finetuned on a dataset acquired by using a top-view configuration. A special loss function called triplet loss was used to train the network. The triplet loss optimizes the embedding space such that data points with the same identity are closer to each other than those with different identities. The TVOW is evaluated on the TVPR2 dataset for people re-ID that is publicly available. The experimental results show that the proposed methods significantly outperform all competitive state-of-the-art methods, bringing to different and significative insights for implicit and extensive shopper behaviour analysis for marketing applications. INDEX TERMS Open-world re-identification, triplet loss, deep neural network, top-view configuration.

show abstract

Section: Resultsmentioning

confidence: 99%

Open-World Person Re-Identification With RGBD Camera in Top-View Configuration for Retail Applications

Martini¹,

Paolanti²,

Frontoni³

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…In [59], the authors started with a two-flow convolutional neural network (CNN) (one for RGB and one for depth) and a final fusion layer. They improved on this approach with a multi-modal attention network [60], adding an attention module to extract local and discriminative features that were fused with globally extracted features. In another work, Lejbolle et al [61] presented a SLATT network with two types of attention modules (one spatial and one layer-wise).…”

Section: Person Re-identificationmentioning

confidence: 99%

Deep understanding of shopper behaviours and interactions using RGB-D vision

Paolanti

Pietrini

Mancini

et al. 2020

Machine Vision and Applications

View full text Add to dashboard Cite

In retail environments, understanding how shoppers move about in a store’s spaces and interact with products is very valuable. While the retail environment has several favourable characteristics that support computer vision, such as reasonable lighting, the large number and diversity of products sold, as well as the potential ambiguity of shoppers’ movements, mean that accurately measuring shopper behaviour is still challenging. Over the past years, machine-learning and feature-based tools for people counting as well as interactions analytic and re-identification were developed with the aim of learning shopper skills based on occlusion-free RGB-D cameras in a top-view configuration. However, after moving into the era of multimedia big data, machine-learning approaches evolved into deep learning approaches, which are a more powerful and efficient way of dealing with the complexities of human behaviour. In this paper, a novel VRAI deep learning application that uses three convolutional neural networks to count the number of people passing or stopping in the camera area, perform top-view re-identification and measure shopper–shelf interactions from a single RGB-D video flow with near real-time performances has been introduced. The framework is evaluated on the following three new datasets that are publicly available: TVHeads for people counting, HaDa for shopper–shelf interactions and TVPR2 for people re-identification. The experimental results show that the proposed methods significantly outperform all competitive state-of-the-art methods (accuracy of 99.5% on people counting, 92.6% on interaction classification and 74.5% on re-id), bringing to different and significative insights for implicit and extensive shopper behaviour analysis for marketing applications.

show abstract

“…To our knowledge, the only previous work to consider multimodal attention is the Multimodal ATtention network (MAT) [28]. In this work, spatial attention weights are calculated for different layers of a CNN based on fusion of features from different modalities.…”

Section: Attention In Person Re-identificationmentioning

confidence: 99%

“…This will be referred to as spatial attention, which is applied in [28] to determine the importance of spatial locations at different layers of a neural network based on fusion of RGB and depth features. Since different layers of a CNN produce features at different abstraction levels [29], features produced by spatial attention modules represent local context information at different abstraction levels.…”

Section: Introductionmentioning

confidence: 99%

Person Re-Identification Using Spatial and Layer-Wise Attention

Lejbolle

Nasrollahi

Krogh³

et al. 2020

IEEE Trans.Inform.Forensic Secur.

Self Cite

View full text Add to dashboard Cite

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.-Users may download and print one copy of any publication from the public portal for the purpose of private study or research.-You may not further distribute the material or use it for any profit-making activity or commercial gain -You may freely distribute the URL identifying the publication in the public portal -

show abstract

Attention in Multimodal Neural Networks for Person Re-identification

Cited by 16 publications

References 31 publications

Open-World Person Re-Identification With RGBD Camera in Top-View Configuration for Retail Applications

Open-World Person Re-Identification With RGBD Camera in Top-View Configuration for Retail Applications

Deep understanding of shopper behaviours and interactions using RGB-D vision

Person Re-Identification Using Spatial and Layer-Wise Attention

Contact Info

Product

Resources

About