Hierarchical Multimodal Adaptive Fusion (HMAF) Network for Prediction of RGB-D Saliency

Abstract

Visual saliency prediction for RGB-D images is more challenging than that for their RGB counterparts. Additionally, very few investigations have been undertaken concerning RGB-D-saliency prediction. The proposed study presents a method based on a hierarchical multimodal adaptive fusion (HMAF) network to facilitate end-to-end prediction of RGB-D saliency. In the proposed method, hierarchical (multilevel) multimodal features are first extracted from an RGB image and depth map using a VGG-16-based two-stream network. Subsequently, the most significant hierarchical features of the said RGB image and depth map are predicted using three two-input attention modules. Furthermore, adaptive fusion of saliencies concerning the above-mentioned fused saliency features of different levels (hierarchical fusion saliency features) can be accomplished using a three-input attention module to facilitate high-accuracy RGB-D visual saliency prediction. Comparisons based on the application of the proposed HMAF-based approach against those of other state-of-the-art techniques on two challenging RGB-D datasets demonstrate that the proposed method outperforms other competing approaches consistently by a considerable margin.

Hand Gesture Recognition Based on Auto-Landmark Localization and Reweighted Genetic Algorithm for Healthcare Muscle Activities

“…The distances that lie vertically, horizontally, and diagonally in the gradient vector are compared with a constant threshold. A distance greater than the threshold is set to the white pixel value of 1, and distances less than the threshold are set to the black pixel value 0 and, as a result, a binary image is formed of outer and internal boundaries of the hand [54]. The resulting directional image is shown in Figure 4.…”

“…The segmented hand is then used for landmark detection. Many approaches are proposed to localize hand landmarks, which help in feature extraction for distinguishing and determining specific gestures [54][55][56][57][58]. The majority of techniques are quite simple and limit the exact localization of landmarks.…”