Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search

Abstract

Text-based person search aims at retrieving target person in an image gallery using a descriptive sentence of that person. It is very challenging since modality gap makes effectively extracting discriminative features more difficult. Moreover, the inter-class variance of both pedestrian images and descriptions is small. Hence, comprehensive information is needed to align visual and textual clues across all scales. Most existing methods merely consider the local alignment between images and texts within a single scale (e.g. only global scale or only partial scale) or simply construct alignment at each scale separately. To address this problem, we propose a method that is able to adaptively align image and textual features across all scales, called NAFS (i.e. Non-local Alignment over Full-Scale representations). Firstly, a novel staircase network structure is proposed to extract full-scale image features with better locality. Secondly, a BERT with localityconstrained attention is proposed to obtain representations of descriptions at different scales. Then, instead of separately aligning features at each scale, a novel contextual non-local attention mechanism is applied to simultaneously discover latent alignments across all scales. The experimental results show that our method outperforms the stateof-the-art methods by 5.53% in terms of top-1 and 5.35% in terms of top-5 on text-based person search dataset. The code is available at https : / / github . com / TencentYoutuResearch/PersonReID-NAFS

DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval

“…Zheng et al [30] propose a Gumbel attention module to alleviate the matching redundancy problem and a hierarchical adaptive matching model is employed to learn subtle feature representations from three different granularities. Recently, the NAFS proposed by Gao et al [5] is designed to extract full-scale image and textual representations with a novel staircase CNN network and a local constrained BERT model. Besides, a multimodal re-ranking algorithm by comparing the visual neighbors of the query to the gallery (RVN) is utilized to further improve the retrieval performance.…”

“…By combining 𝐴𝑙𝑖𝑔𝑛𝐼𝐼 and 𝐴𝑙𝑖𝑔𝑛𝐼𝐼𝐼 with 𝐴𝑙𝑖𝑔𝑛𝐼 , SPSM can more completely separate person and surroundings information with the aid of SPFM and PDM. Besides, comparing the third row from the bottom with the last row in Table 1, the top-1, top-5 and top-10 performance increase by 1.12%, 0.71%, 0.61% and 1.43, [23] 13.66 -41.72 GNA-RNN [12] 19.05 -53.64 IATV [11] 25.94 -60.48 PWM-ATH [3] 27.14 49.45 61.02 Dual Path [31] 44.40 66.26 75.07 GLA [2] 43.58 66.93 76.26 MIA [18] 53.10 75.00 82.90 A-GANet [14] 53.14 74.03 81.95 GALM [8] 54.12 75.45 82.97 TIMAM [17] 54.51 77.56 84.78 IMG-Net [25] 56.48 76.89 85.01 CMAAM [1] 56.68 77.18 84.86 HGAN [30] 59.00 79.49 86.6 NAFS [5] 59 4, ablation experiments are conducted to search for the optimal zeroing rate 𝑟 . It can be observed that initially the performance of DSSL follows a increasing tendency with the growth of 𝑟 .…”

Text-based Person Search in Full Images via Semantic-Driven Proposal Generation

“…Although the multi-scale alignment provide supplement for global feature matching, the alignment for each scale is fixed. Gao et al [33] realized the need to align visual and textual clues across all scales and proposed cross-scale alignment for text-based person search.…”

“…Then both the visual feature and semantic feature from the two paths are fed into proposal-text embedding module. Inspired by [33], we propose a RoI-level cross scale matching scheme which utilizes mixed multi-scale features extracted from person proposals and text descriptions for feature embedding with the help of the nonlocal attention mechanism. Besides, the CMPM and CMPC loss are enforced on top of the global features to supervise the proposal-text cross-modal feature embedding process.…”

“…Although the text-based person retrieval task (Figure 1(c)), which aims to match a given text query with a gallery of cropped person images, has been explored in recent years [33]- [42]. However, there is still a step distance from the real-world scenarios as the bounding box annotations are unavailable and the query-described person needs to be searched in a gallery of full images.…”