Text-based Person Search in Full Images via Semantic-Driven Proposal Generation

Abstract

Finding target persons in full scene images with a query of text description has important practical applications in intelligent video surveillance. However, different from the realworld scenarios where the bounding boxes are not available, existing text-based person retrieval methods mainly focus on the cross modal matching between the query text descriptions and the gallery of cropped pedestrian images. To close the gap, we study the problem of text-based person search in full images by proposing a new end-to-end learning framework which jointly optimize the pedestrian detection, identification and visual-semantic feature embedding tasks. To take full advantage of the query text, the semantic features are leveraged to instruct the Region Proposal Network to pay more attention to the text-described proposals. Besides, a cross-scale visual-semantic embedding mechanism is utilized to improve the performance. To validate the proposed method, we collect and annotate two large-scale benchmark datasets based on the widely adopted image-based person search datasets CUHK-SYSU and PRW. Comprehensive experiments are conducted on the two datasets and compared with the baseline methods, our method achieves the state-of-the-art performance.

Feature semantic alignment and information supplement for Text-based person search

“…In recent years researchers have proposed a variety of methods, which can be broadly classified into the following three categories: similarity relation metrics based methods Niu et al [19]; Gao et al [20]; Li et al [30], external knowledge assistance based methods Wang et al [16]; Jing et al [15]; Aggarwal et al [17]; Wang et al [18], and multi-granularity relational correspondence based feature alignment methods Ding et al [10]; Zhang et al [11]; Liu et al [12]; Zheng et al [13]; Chen et al [14]; Wang et al [21]. Similarity relation metrics based methods use the similarity between text features and image features as the relationship between them to obtain robust features.…”

“…In recent years, many effective methods of feature semantic consistency have been proposed. These methods can be roughly divided into methods based on local relationship correspondence Ding et al [10]; Zhang et al [11]; Liu et al [12]; Zheng et al [13]; Chen et al [14], etc. ), methods based on external knowledge Jing et al [15]; Wang et al [16]; Aggarwal et al [17]; Wang et al [18], methods based on similarity measurement Niu et al [19]; Gao et al [20], and methods based on multi-head attention mechanism Wang et al [21]; Li et al [22].…”