Incorporating Luminance, Depth and Color Information by a Fusion-Based Network for Semantic Segmentation

Hung, Shang-Wei; Lo, Shao-Yuan; Hang, Hsueh-Ming

doi:10.1109/icip.2019.8803360

Cited by 39 publications

(35 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Related applications in multimodal fusion networks can be found in [69,70,71]. Also, the 1 × 1 convolution layer is commonly used to allow complex and learnable interaction across modalities and channels [72,70]. Besides, attention mechanism has become a powerful tool for image recognition [73,74,75].…”

Section: Semantic Image Segmentationmentioning

confidence: 99%

See 1 more Smart Citation

Deep multimodal fusion for semantic image segmentation: A survey

Zhang

Sidibé

Morel

et al. 2021

Image and Vision Computing

144

View full text Add to dashboard Cite

Recent advances in deep learning have shown excellent performance in various scene understanding tasks. However, in some complex environments or under challenging conditions, it is necessary to employ multiple modalities that provide complementary information on the same scene. A variety of studies have demonstrated that deep multimodal fusion for semantic image segmentation achieves significant performance improvement. These fusion approaches take the benefits of multiple information sources and generate an optimal joint prediction automatically. This paper describes the essential background concepts of deep multimodal fusion and the relevant applications in computer vision. In particular, we provide a systematic survey of multimodal fusion methodologies, multimodal segmentation datasets, and quantitative evaluations on the benchmark datasets. Existing fusion methods are summarized according to a common taxonomy: early fusion, late fusion, and hybrid fusion. Based on their performance, we analyze the strengths and weaknesses of different fusion strategies. Current challenges and design choices are discussed, aiming to provide the reader with a comprehensive and heuristic view of deep multimodal image segmentation.

show abstract

Section: Semantic Image Segmentationmentioning

confidence: 99%

“…Hung et al [72] presented LDFNet that contains a well-designed encoder for the non-RGB branch, aiming to fully make use of luminance, depth, and color information. Recently, RFBNet [105] was proposed with an efficient fusion mechanism that explores the interdependence between the encoders (see Figure 5).…”

Section: Early Fusionmentioning

confidence: 99%

Deep multimodal fusion for semantic image segmentation: A survey

Zhang

Sidibé

Morel

et al. 2021

Image and Vision Computing

144

View full text Add to dashboard Cite

show abstract

“…In (c) as proposed in ACNet [34] and RFBNet [35], in addition to paths for each modality, an additional path for feature fusion (i.e., a path colored by magenta in the figure) is employed. For the case where one of the input modalities is equal to the output modality, in (d) [36], [37], only one modality is regarded as dominant. For intensive analysis of this dominant modality, features extracted from other modalities are auxiliarily fused to those extracted from the dominant modality.…”

Section: A Insar Analysis and Its Extensionsmentioning

confidence: 99%

Multi-Modal Data Fusion for Land-Subsidence Image Improvement in PSInSAR Analysis

Shimosato

Ukita

2021

IEEE Access

View full text Add to dashboard Cite

There are three popular methods to understand the land subsidence: leveling, Global Navigation Satellite System, and Interferometric Synthetic Aperture Radar (InSAR) analysis using SAR images. While both leveling and the Global Navigation Satellite System can measure the amount of land subsidence only at specific points, InSAR analysis can observe a wide area in short time intervals. In terms of accuracy, however, InSAR analysis is inferior to leveling; centimeter/millimeter order (InSAR/PSInSAR analysis) vs. millimeter order (leveling). Among all observation errors in InSAR analysis, a tropospheric delay error has a large adverse effect on the measurement. It is difficult to suppress this tropospheric delay error by conventional methods because they try to remove error at each pixel independently in an InSAR image. However, geometrically-neighboring regions/pixels should be naturally correlated. Our proposed method employs such a neighboring relationship in a convolutional neural network (CNN). Our CNN is designed to improve InSAR analysis by mutually incorporating the InSAR image and the tropospheric delay error, which are estimated by any conventional methods. Experimental results demonstrate that our proposed method can reduce the mean error compared with a conventional method: from 10.3mm to 6.80mm. INDEX TERMS PSInSAR analysis, Tropospheric delay, Deep convolutional neural networks, Multi-modal data fusion I. INTRODUCTIONThe ground always shifts vertically and horizontally. One of the critical shifts is a land subsidence. Once the land subsidence occurs, the ground hardly returns to its original state. It is essential to understand the land subsidence and to take a countermeasure against it as early as possible, because it may cause huge negative impacts on our lives such as building collapse and lifeline damages.We have three popular methods to measure the land subsidence: leveling, Global Navigation Satellite Systems (GNSSs), and Interferometric Synthetic Aperture Radar (In-SAR) analysis using SAR images. The functional properties of these three methods are summarized in Table 1. For leveling, a huge human effort is required in order to measure a wide area. The GNSSs including Global Positioning System (GPS), GLObal NAvigation Satellite System (GLONASS), Galileo, and Quasi-Zenith Satellite System (QZSS) measure the crustal alteration so that ground-installation devices receive signals transmitted from artificial satellites. SAR is a form of high spatial-resolution micro-wave radar. SAR is transmitted from a satellite towards the ground surface. Its 21 reflection intensity and phase are captured by the same satel-22 lite. From the phase shift between those captured at different 23 capturing times, we can compute the ground subsidence. As 24 summarized in Table 1, while both the leveling and GNSS 25 (1) require special measurement devices and (2) measure the 26 amount of the land subsidence only at specific points, InSAR 27 analysis can observe a wide area in short time intervals with 28 no ground-installat...

show abstract

“…These contributions use a large dataset such as ImageNet [35] for pre-trained models. Recent segmentation techniques have distinct characteristics denoted by their design such as: (1) network topology: pooling indices [3], skip connection [34], multi-path refinement [27], pyramid pooling [47], fusionbased architecture [15] and dense connectivity [20], (2) varying input: colour RGB or RGB-D with depth [15], [19], depth and luminance [21], and illumination invariance [1], and (3) consideration of adverse-weather conditions [8], [14], [36]. As the main objective of this work is semantic scene segmentation under foggy weather conditions, recent studies in this specific domain are specifically presented in this section.…”

Section: A Semantic Segmentationmentioning

confidence: 99%

“…Despite the general trend of performance improvement within automotive scene understanding [4], [17], [27], [47], there is still significant room for improvement across the spectrum of non-ideal operating conditions. In parallel with using recent image segmentation techniques [15], [20], [21], [40], employing the concept of image-to-image translation to map one domain onto another [22], [48] is a useful step that enables accurate semantic segmentation performance under extreme weather conditions.…”

Section: Introductionmentioning

confidence: 99%

Multi-Modal Learning for Real-Time Automotive Semantic Foggy Scene Understanding via Domain Adaptation

Alshammari

Akçay

Breckon

2021

2021 IEEE Intelligent Vehicles Symposium (IV)

View full text Add to dashboard Cite

Robust semantic scene segmentation for automotive applications is a challenging problem in two key aspects: (1) labelling every individual scene pixel and (2) performing this task under unstable weather and illumination changes (e.g., foggy weather), which results in poor outdoor scene visibility. Such visibility limitations lead to non-optimal performance of generalised deep convolutional neural network-based semantic scene segmentation. In this paper, we propose an efficient endto-end automotive semantic scene understanding approach that is robust to foggy weather conditions. As an end-to-end pipeline, our proposed approach provides: (1) the transformation of imagery from foggy to clear weather conditions using a domain transfer approach (correcting for poor visibility) and ( 2) semantically segmenting the scene using a competitive encoder-decoder architecture with low computational complexity (enabling real-time performance). Our approach incorporates RGB colour, depth and luminance images via distinct encoders with dense connectivity and features fusion to effectively exploit information from different inputs, which contributes to an optimal feature representation within the overall model. Using this architectural formulation with dense skip connections, our model achieves comparable performance to contemporary approaches at a fraction of the overall model complexity.

show abstract

Incorporating Luminance, Depth and Color Information by a Fusion-Based Network for Semantic Segmentation

Cited by 39 publications

References 13 publications

Deep multimodal fusion for semantic image segmentation: A survey

Deep multimodal fusion for semantic image segmentation: A survey

Multi-Modal Data Fusion for Land-Subsidence Image Improvement in PSInSAR Analysis

Multi-Modal Learning for Real-Time Automotive Semantic Foggy Scene Understanding via Domain Adaptation

Contact Info

Product

Resources

About