Causal Reasoning Meets Visual Representation Learning: A Prospective Study

Abstract

Visual representation learning is ubiquitous in various real-world applications, including visual comprehension, video understanding, multi-modal analysis, human-computer interaction, and urban computing. Due to the emergence of huge amounts of multimodal heterogeneous spatial/temporal/spatial-temporal data in the big data era, the lack of interpretability, robustness, and out-of-distribution generalization are becoming the challenges of the existing visual models. The majority of the existing methods tend to fit the original data/variable distributions and ignore the essential causal relations behind the multi-modal knowledge, which lacks unified guidance and analysis about why modern visual representation learning methods easily collapse into data bias and have limited generalization and cognitive abilities. Inspired by the strong inference ability of human-level agents, recent years have therefore witnessed great effort in developing causal reasoning paradigms to realize robust representation and model learning with good cognitive ability. In this paper, we conduct a comprehensive review of existing causal reasoning methods for visual representation learning, covering fundamental theories, models, and datasets. The limitations of current methods and datasets are also discussed. Moreover, we propose some prospective challenges, opportunities, and future research directions for benchmarking causal reasoning algorithms in visual representation learning. This paper aims to provide a comprehensive overview of this emerging field, attract attention, encourage discussions, bring to the forefront the urgency of developing novel causal reasoning methods, publicly available benchmarks, and consensus-building standards for reliable visual representation learning and related real-world applications more efficiently.

A progressive segmentation with weight contrast label enhancement for weakly supervised video salient object detection

“…The ability to judge the goodness of received information enables us to select positive information while eliminating negative ones. This concept has been applied to various computer vision tasks [28][29][30].…”

Unsupervised Domain Adaptation Semantic Segmentation of High-Resolution Remote Sensing Imagery With Invariant Domain-Level Prototype Memory

“…For explaining black-box visual classifiers, [56] formulated a causal extension to the paradigm of instance-wise features selection and obtain a subset of input features that has the greatest causal effect on the model's output. [57] conducted a comprehensive review of existing causal reasoning methods for visual representation learning and pointed out the importance of causal reasoning in visual representation learning.…”

Pedestrian re-identification domain generalization algorithm based on causal strong and weak alignment

“…Traditional feature learning methods are prone to learning pseudo-correlation properties introduced by confounding factors, which is not conducive to model generalisation across domains 11 . Causal inference can eliminate this pseudocorrelation by replacing the conditional distribution with an intervening distribution 11 .…”

“…Traditional feature learning methods are prone to learning pseudo-correlation properties introduced by confounding factors, which is not conducive to model generalisation across domains 11 . Causal inference can eliminate this pseudocorrelation by replacing the conditional distribution with an intervening distribution 11 . In the image task, Wang 12 proposed a causal attention module to help deep models learn causal features with robustness by annotating contextual information obfuscation factors in an unsupervised manner.…”