Inverted Pyramid Multi-task Transformer for Dense Scene Understanding

Hanrong, Ye,; Xu, Dan

doi:10.1007/978-3-031-19812-0_30

Cited by 42 publications

(12 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our model outperforms all the multitask baselines, i.e. ST-MTL [36], InvPT [70], Taskprompter [71], and MulT [3], respectively. For instance, our model correctly segments and predicts the surface normal of the elements within the yellow-circled region, unlike the baseline.…”

Section: Additional Qualitative Resultsmentioning

confidence: 88%

“…InvPT [70]: performs simultaneous modeling of spatial positions and multiple dense prediction tasks in a unified transformer framework.…”

Section: Transformer-based Methodsmentioning

confidence: 99%

“…Taskprompter [71]: focuses on the representation learning capability of the multitask networks by using a set of task prompts.…”

Section: Transformer-based Methodsmentioning

confidence: 99%

See 2 more Smart Citations

MuIT: An End-to-End Multitask Learning Transformer

Bhattacharjee

Zhang

Süsstrunk

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We introduce the first multitasking vision transformer adapters that learn generalizable task affinities which can be applied to novel tasks and domains. Integrated into an off-the-shelf vision transformer backbone, our adapters can simultaneously solve multiple dense vision tasks in a parameter-efficient manner, unlike existing multitasking transformers that are parametrically expensive. In contrast to concurrent methods, we do not require retraining or fine-tuning whenever a new task or domain is added. We introduce a task-adapted attention mechanism within our adapter framework that combines gradient-based task similarities with attention-based ones. The learned task affinities generalize to the following settings: zero-shot task transfer, unsupervised domain adaptation, and generalization without fine-tuning to novel domains. We demonstrate that our approach outperforms not only the existing convolutional neural network-based multitasking methods but also the vision transformer-based ones. Our project page is at https://ivrl.github.io/VTAGML.

show abstract

Section: Additional Qualitative Resultsmentioning

confidence: 88%

“…InvPT [70]: performs simultaneous modeling of spatial positions and multiple dense prediction tasks in a unified transformer framework.…”

Section: Transformer-based Methodsmentioning

confidence: 99%

See 1 more Smart Citation

MuIT: An End-to-End Multitask Learning Transformer

Bhattacharjee

Zhang

Süsstrunk

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Other Efforts to Unify Dense Prediction Tasks Realizing segmentation, depth and surface normal are all pixelwise mapping problem, previous works [10,52,67,72,96] have deployed them to the same framework. Some works [11,12,103,104,106] improve performance by exploring the relations among dense prediction tasks. UViM [48] and Painter [96] unify segmentation and depth estimation, but neither of them is based on discretizing the continuous output space, and neither considers surface normal estimation.…”

Section: Related Workmentioning

confidence: 99%

Porosity and permeability prediction using a transformer and periodic long short-term network

et al. 2023

View full text Add to dashboard Cite

Effective reservoir parameter prediction is important for subsurface characterization and understanding the fluid migration. However, conventional methods for obtaining porosity and permeability are based on either core measurements or mathematical/petrophysical modeling, which are expensive or inefficient. In this study, we propose a reliable and low-cost deep learning (DL) framework for reservoir permeability and porosity prediction from real logging data at different regions. We leverage an advanced learning architecture (i.e., the transformer model) and design a new regression network (RPTransformer) that is sensitive to the depth period change of the logging data. The RPTransformer is composed of 1-D convolutional, long- and short-term memory (LSTM), and transformer layers. First, we use a 1-D convolutional layer for the first layer of the network to extract the significant features from logging data. Then, the nonlinear mapping relationships between logging data and reservoir parameters are established using several LSTM layers with a period parameter. Afterward, we use the encoder in the vision transformer (ViT) with the self-attention mechanism to further extract logging data features. The proposed network is a data-driven supervised learning framework and shows highly accurate and robust prediction results when applied to different geographic regions. To demonstrate the reliable prediction performance of our proposed network, we compare it with several classic machine learning and state-of-the-art DL methods, e.g., random forest (RF), multilayer LSTM (MLSTM), and LSTNet. More importantly, we show the generalization and uncertainty of the network in real-world applications through comprehensive numerical experiments.

show abstract

“…Neural Networks (CNN). Several techniques have made significant advancements in extracting deep information, whether supervised [7], [8], [9], [10], semi-supervised [11], [12], [13] or unsupervised [14], [15], [16]. The first impressive single image depth estimation based on CNN, Eigen et al [7] estimated depth information using two independent deep neural networks.…”

Section: Various Approaches Have Accomplished Remarkable Improvements...mentioning

confidence: 99%

TP-GAN: Simple Adversarial Network With Additional Player for Dense Depth Image Estimation

Hendra

Kanazawa

2023

IEEE Access

View full text Add to dashboard Cite

We present a simple yet robust monocular depth estimation technique by synthesizing a depth map image from a single RGB input image using the advantage of generative adversarial networks (GAN). We employ an additional sub-model termed refiner to extract local depth features, then combine it with the global scene information from the generator to improve the GAN's performance compared to the standard GAN architectural scheme. Notably, the generator is the first player to learn to synthesize depth images. The second player, the discriminator, classifies the generated depth. In the meantime, the third player, the refiner, enhances the final reconstructed depth. Complementing the GAN model, we apply a conditional generative network (cGAN) to lead the generator in mapping the input image to the respective depth representation. We further incorporate a structured similarity (SSIM) as our loss function for the generator and refiner in GAN training. Through extensive experiment validation, we confirmed the performance of our strategy on the publicly indoor NYU Depth v2 and KITTI outdoor data. Experiment results on the NYU depth v2 dataset show that our proposed approach achieves the best performance by 96.0% on threshold accuracy (δ < 1.25 2 ) and the second-best accuracy on all thresholds on the KITTI dataset. We discovered that our proposed method compares favorably to numerous existing monocular depth estimation strategies and demonstrates a considerable improvement in the accuracy of image depth estimation despite its simple network architecture.INDEX TERMS Depth estimation, single image, conditional GAN, generative adversarial network (GAN), third player GAN.

show abstract

Inverted Pyramid Multi-task Transformer for Dense Scene Understanding

Cited by 42 publications

References 54 publications

MuIT: An End-to-End Multitask Learning Transformer

MuIT: An End-to-End Multitask Learning Transformer

Porosity and permeability prediction using a transformer and periodic long short-term network

TP-GAN: Simple Adversarial Network With Additional Player for Dense Depth Image Estimation

Contact Info

Product

Resources

About