Multi-Modal Visual Place Recognition in Dynamics-Invariant Perception Space

Abstract

Visual place recognition is one of the essential and challenging problems in the fields of robotics. In this letter, we for the first time explore the use of multi-modal fusion of semantic and visual modalities in dynamics-invariant space to improve place recognition in dynamic environments. We achieve this by first designing a novel deep learning architecture to generate the static semantic segmentation and recover the static image directly from the corresponding dynamic image. We then innovatively leverage the spatial-pyramid-matching model to encode the static semantic segmentation into feature vectors. In parallel, the static image is encoded using the popular Bag-ofwords model. On the basis of the above multi-modal features, we finally measure the similarity between the query image and target landmark by the joint similarity of their semantic and visual codes. Extensive experiments demonstrate the effectiveness and robustness of the proposed approach for place recognition in dynamic environments.

Brain-inspired multimodal hybrid neural network for robot place recognition

“…It refers to the problem of deciding whether a place has been visited before, and, if it has been visited before, which place it was (1). Many recent studies have achieved great progress in improving the performance of place recognition with a single image or jointly with image, geometric, and semantic information (1)(2)(3)(4)(5). However, place recognition in natural environments remains a huge challenge because of rapid environmental changes and stringent requirements for power, computing, and latency caused by the limited resources of robots.…”

“…In contrast, humans and animals demonstrate remarkable place recognition capabilities, robustly identifying places in large threedimensional environments (23)(24)(25)(26). They can reliably sense, robustly represent, and efficiently recognize places using an internal conjunctive representation of spatial view cells (27,28), auditory view cells (3,(29)(30)(31), olfactory view cells (32)(33)(34), place cells (35,36), head direction cells (37,38), grid cells (39,40), and time cells (41)(42)(43). As illustrated in Fig.…”

Framework Construction of an Adversarial Federated Transfer Learning Classifier

“…In this context, federated learning (FL) is gaining popularity because of its capacity to provide collaborative training while maintaining data privacy, as well as a solution to the problem of isolated data islands [33]. The end-to-end diagnostic framework with automatic feature extraction may be simply established when the training and testing data are from the same distribution [12] [31]. However, the distributions of datasets in the real-world medical diagnosis industry vary by domain.…”

An Outdoor Pedestrian Localization Scheme Fusing PDR and VPR

“…VPR is to process the visual information, extract the corresponding geographic location information, integrate the information, judge whether it is within the recognition category, and give the ranking of similarity. In recent years, with the development of deep learning technology, VPR has made great progress in the fields of location recognition, mobile robot, virtual reality, and enhancement [13][14][15]. However, it is disturbed by environmental factors such as illumination, and the localization of a single point is much better than continuous long-distance distance.…”