Budget-Aware Deep Semantic Video Segmentation

Mahasseni, Behrooz; Todorovic, Sinisa; Fern, Alan

doi:10.1109/cvpr.2017.224

Cited by 50 publications

(36 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, temporal dynamics are also learnt using a sequential model like LSTM [52]. Moreover, LSTM is also used to select keyframes for video scene parsing [32]. Bowen Wang et al [48], proposed Noisy-LSTM which uses convLSTM for video semantic segmentation.…”

Section: Related Workmentioning

confidence: 99%

UVid-Net: Enhanced Semantic Segmentation of UAV Aerial Videos by Embedding Temporal Information

Girisha

Verma

Pai

et al. 2021

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Semantic segmentation of aerial videos has been extensively used for decision making in monitoring environmental changes, urban planning, and disaster management. The reliability of these decision support systems is dependent on the accuracy of the video semantic segmentation algorithms. The existing CNN based video semantic segmentation methods have enhanced the image semantic segmentation methods by incorporating an additional module such as LSTM or optical flow for computing temporal dynamics of the video which is a computational overhead. The proposed research work modifies the CNN architecture by incorporating temporal information to improve the efficiency of video semantic segmentation.In this work, an enhanced encoder-decoder based CNN architecture (UVid-Net) is proposed for UAV video semantic segmentation. The encoder of the proposed architecture embeds temporal information for temporally consistent labelling. The decoder is enhanced by introducing the feature-refiner module, which aids in accurate localization of the class labels. The proposed UVid-Net architecture for UAV video semantic segmentation is quantitatively evaluated on extended ManipalUAVid dataset. The performance metric mIoU of 0.79 has been observed which is significantly greater than the other state-of-the-art algorithms. Further, the proposed work produced promising results even for the pre-trained model of UVid-Net on urban street scene with fine tuning the final layer on UAV aerial videos.

show abstract

Section: Related Workmentioning

confidence: 99%

UVid-Net: Enhanced Semantic Segmentation of UAV Aerial Videos by Embedding Temporal Information

Girisha

Verma

Pai

et al. 2021

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

show abstract

“…The Flow-Track framework [45] combines historical feature flows, warping, consine similarity and temporal attention for efficient feature aggregation achieving an increased performance. Some techniques, primarily meant for semantic object segmentation [46]- [48], also utilized warped feature flow maps to aid the final stages of the segmentation pipeline.…”

Section: B Feature Aggregation Over Timementioning

confidence: 99%

“…Other techniques take advantage of the reinforcement learning framework for adaptive keyframe selection [48], [50], [61]. A policy-gradient reinforcement-learning approach [48] makes budget-aware processing by approximating the gradient of a non-decomposable and nondifferentiable objective. In [68], a learned policy is able to select when to update or to initiate a tracker.…”

Section: Adaptive Keyframe Selectionmentioning

confidence: 99%

“…While prior works, mostly flow [40]- [48] and recurrent [44], [49]- [53] methods, also provide approaches for fast video object detection with fast and slow detection stages, these approaches are based on processing the full image for every frame, incurring in a redundancy of computation and not exploiting the performance increase when the object appears reduced in the image. Conversely, our method takes advantage of the redundancy of context in video frames, increasing the performance when objects appear small in the image.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Adaptive Inattentional Framework for Video Object Detection With Reward-Conditional Training

et al. 2020

View full text Add to dashboard Cite

Recent object detection studies have been focused on video sequences, mostly due to the increasing demand of industrial applications. Although single-image architectures achieve remarkable results in terms of accuracy, they do not take advantage of particular properties of the video sequences and usually require high parallel computational resources, such as desktop GPUs. In this work, an inattentional framework is proposed, where the object context in video frames is dynamically reused in order to reduce the computation overhead. The context features corresponding to keyframes are fused into a synthetic feature map, which is further refined using temporal aggregation with ConvLSTMs. Furthermore, an inattentional policy has been learned to adaptively balance the accuracy and the amount of context reused. The inattentional policy has been learned under the reinforcement learning paradigm, and using our novel reward-conditional training scheme, which allows for policy training over a whole distribution of reward functions and enables the selection of a unique reward function at inference time. Our framework shows outstanding results on platforms with reduced parallelization capabilities, such as CPUs, achieving an average latency reduction up to 2.09x, and obtaining FPS rates similar to their equivalent GPU platform, at the cost of a 1.11x mAP reduction.

show abstract

“…Moreover, as a sequential learning network, recurrent neural network (RNN) [23] or long short‐term memory (LSTM) [24] is also utilised to adapt CNN‐based image segmentation methods to video semantic segmentation [8]. In [25], they propose a budget‐aware framework that learns to optimally select a small subset of frames for pixel‐wise labelling by a CNN, and then interpolates the obtained segmentations to unprocessed frames. However, the method first needs to use an image‐based method to segment some video frames.…”

Section: Related Workmentioning

confidence: 99%

Capturing the spatio‐temporal continuity for video semantic segmentation

Chen

Han

2019

IET image process

View full text Add to dashboard Cite

In recent years, image semantic segmentation based on a convolutional neural network has achieved many advances. However, the development of video semantic segmentation is relatively slow. Directly applying the image segmentation algorithms to each video frame separately may ignore the temporal region continuity inherent in videos. In this study, the authors propose a novel deep neural network architecture with a newly devised spatio‐temporal continuity (STC) module for video semantic segmentation. Particularly, the architecture includes an encoding network, an STC module, and a decoding network. The encoding network is used to extract a high‐level feature map. The STC module then uses the high‐level feature map as input to extract the STC feature map. For decoding, they use four dilated convolutional layers to obtain more abstract representation and a deconvolutional layer to increase the size of the representation. Finally, they fuse the current feature representation and the previous feature representation and get the class probabilities. Thus, this architecture receives a sequence of consecutive video frames and outputs the segmentation result of the current frame. They extensively evaluate the proposed approach on the CamVid and KITTI datasets. Compared with other methods, the authors’ approach not only achieves competitive performance but also has lower complexity.

show abstract

Budget-Aware Deep Semantic Video Segmentation

Cited by 50 publications

References 27 publications

UVid-Net: Enhanced Semantic Segmentation of UAV Aerial Videos by Embedding Temporal Information

UVid-Net: Enhanced Semantic Segmentation of UAV Aerial Videos by Embedding Temporal Information

Adaptive Inattentional Framework for Video Object Detection With Reward-Conditional Training

Capturing the spatio‐temporal continuity for video semantic segmentation

Contact Info

Product

Resources

About