Mining Relations Among Cross-Frame Affinities for Video Semantic Segmentation

Sun, Guolei; Liu, Yun; Tang, Hao; Chhatkuli, Ajad; Zhang, Le; Gool, Luc Van

doi:10.1007/978-3-031-19830-4_30

Cited by 18 publications

(2 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given they can't provide the instance mask for each building, the prediction results also need to be post-processed. 3) video semantic segmentation methods: CFFM [13] and MRCFA [14] are adopted since they can process the multi-temporal image series which is similar to the video frames. To ensure fair comparisons, we employ the same post-processing algorithm in conjunction with the compared methods.…”

Section: Compared With the State-of-the-artmentioning

confidence: 99%

BuildMon: Building Extraction and Change Monitoring in Time Series Remote Sensing Images

Wang,

Chen,

Zhang

et al. 2024

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Building extraction and change monitoring in remote sensing (RS) imagery play pivotal roles in various applications, including urban planning, disaster management, and infrastructure monitoring. While significant progress has been made in single and bi-temporal RS images, effectively harnessing the rich temporal information of time series RS images remains a challenge. Time series RS images offer an extended temporal span for monitoring dynamic changes in building instances. However, they often exhibit noticeable appearance discrepancies and feature variations, presenting substantial obstacles to effective multitemporal information aggregation. To address these challenges, we introduce a Building Extraction and Change Monitoring Network (abbreviated as BuildMon), which jointly explores the segmentation masks, location tracking, and construction status of building instances. Our approach incorporates a spatial-temporal transformer to model relationships between images at different time spans. The windowed attention module within it can capture spatial-temporal context for a larger scope of feature aggregation. For enhancing the performance on both tasks, we adopted ground truth masks and semantic change information together as supervisory signals for BuildMon. This is complemented by the specially designed change-guided loss function, which specifically highlights regions of change and assigns targeted weights to building areas within the imagery. To validate the effectiveness of our method, we conduct comprehensive experiments on the SpaceNet 7 dataset. The results showcase the state-of-the-art performance of our approach, achieving mIoU and SCOT metrics of 67.90 and 39.73, respectively.

show abstract

Section: Compared With the State-of-the-artmentioning

confidence: 99%

BuildMon: Building Extraction and Change Monitoring in Time Series Remote Sensing Images

Wang,

Chen,

Zhang

et al. 2024

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

show abstract

“…The work of [53] has devised a spatio‐temporal continuity module to capture the spatial and temporal relations inherent in videos to improve the segmentation performance. In [61], better temporal information aggregation in videos has been achieved by mining relations among cross‐frame affinities instead of developing new techniques to calculate the cross‐frame affinities such as optical flow and attention. To reduce the computation cost, some works apply a strong segmentation network only at predefined key frames, and propagate to non‐key frames using optical flows to reuse the better features in preceding frames [39, 43–45].…”

Section: Related Workmentioning

confidence: 99%

TRACL: Temporal reconstruction and adaptive consistency loss for semi‐supervised video semantic segmentation

Liang,

Dong,

Zhang

2023

IET Image Processing

View full text Add to dashboard Cite

While existing supervised semantic segmentation methods have shown significant performance improvements, they heavily rely on large‐scale pixel‐level annotated data. To reduce this dependence, recent research has proposed semi‐supervised learning‐based methods that have achieved great success. However, almost all these works are mainly dedicated to image semantic segmentation, while semi‐supervised video semantic segmentation (SVSS) has been barely explored. Due to the significant difference between video data and image, simply adapting semi‐supervised image semantic segmentation approaches to SVSS may neglect the inherent temporal correlations in video frames. This paper presents a novel method (named TRACL) with temporal reconstruction (TR) and adaptive consistency loss (ACL) for SVSS, aiming to fully utilize the temporal relations of internal frames in video clip. The authors’ TR method implements the reconstruction from the feature and output levels to narrow the distribution gap between internal video frames. Specifically, considering the underlying data distribution, the authors construct Gaussian models for each category, and use probability density function to obtain the similarity between different feature maps for temporal feature reconstruction. The authors’ ACL can adaptively select two pixel‐wise consistency loss including Flow Consistency Loss and Reconstruction Consistency Loss, providing stronger supervision signals for unlabelled frames during model training. Additionally, the authors extend their method to unlabelled video for more training data by employing mean‐teacher structure. Extensive experiments on three datasets including Cityscapes, Camvid and VSPW demonstrate that the authors’ proposed method outperforms previous state‐of‐the‐art methods.

show abstract