MuIT: An End-to-End Multitask Learning Transformer

Bhattacharjee, Deblina; Zhang, Tong; Süsstrunk, Sabine; Salzmann, Mathieu

doi:10.1109/cvpr52688.2022.01172

Cited by 45 publications

(32 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, this limits the performance of dense prediction tasks whose performances are significantly influenced by the resolution of the feature maps generated by the model. Simultaneous to our work, several studies have investigated multi-task transformers [15], [16], [17]. However, their attention maps still possess limited context and lack the capability to model spatial and cross-task interactions globally among features of different tasks.…”

Section: Global Spatial Interaction Simultaneous All-task Interactionmentioning

confidence: 99%

“…MTFormer [16] and MQTransformer [17] develop different cross-task information fusion modules, yet the multi-task information cannot interact with each other in a spatially global context. Similar to MTAN [26], the task-specific modules of MulT [15] directly query the task-shared backbone feature to acquire task-specific features but are unable to model cross-task interactions. MultiMAE [34] learns a versatile backbone model but requires fine-tuning on each single task, making it essentially a series of single-task models instead of one multi-task model.…”

Section: Multi-task Learning For Dense Scene Understandingmentioning

confidence: 99%

See 1 more Smart Citation

Inverted Pyramid Multi-task Transformer for Dense Scene Understanding

Hanrong

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Multi-task scene understanding aims to design models that can simultaneously predict several scene understanding tasks with one versatile model. Previous studies typically process multi-task features in a more local way, and thus cannot effectively learn spatially global and cross-task interactions, which hampers the models' ability to fully leverage the consistency of various tasks in multitask learning. To tackle this problem, we propose an Inverted Pyramid multi-task Transformer, capable of modeling cross-task interaction among spatial features of different tasks in a global context. Specifically, we first utilize a transformer encoder to capture task-generic features for all tasks. And then, we design a transformer decoder to establish spatial and cross-task interaction globally, and a novel UP-Transformer block is devised to increase the resolutions of multi-task features gradually and establish cross-task interaction at different scales. Furthermore, two types of Cross-Scale Self-Attention modules, i.e., Fusion Attention and Selective Attention, are proposed to efficiently facilitate cross-task interaction across different feature scales. An Encoder Feature Aggregation strategy is further introduced to better model multi-scale information in the decoder. Comprehensive experiments on several 2D/3D multi-task benchmarks clearly demonstrate our proposal's effectiveness, establishing significant state-of-the-art performances. The codes are publicly available here.

show abstract

Section: Global Spatial Interaction Simultaneous All-task Interactionmentioning

confidence: 99%

Section: Multi-task Learning For Dense Scene Understandingmentioning

confidence: 99%

Inverted Pyramid Multi-task Transformer for Dense Scene Understanding

Hanrong

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Our DeMT obtains 46.36 SemSeg accuracy, which is 6.3% higher than that of MQTransformer with the same Swin-T backbone and slightly lower FLOPs (100.7G vs. 106.02G). MuIT (Bhattacharjee et al 2022) reports a 13.3% and 8.54% increase in relative performance for semantic…”

Section: Comparison With the State-of-the-artmentioning

confidence: 99%

DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction

Xu¹,

Yang²,

Zhang³

2023

Preprint

View full text Add to dashboard Cite

Convolution neural networks (CNNs) and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL). Most of the current studies on MTL solely rely on CNN or Transformer. In this work, we present a novel MTL model by combining both merits of deformable CNN and query-based Transformer for multi-task learning of dense prediction. Our method, named DeMT, is based on a simple and effective encoder-decoder architecture (i.e., deformable mixer encoder and task-aware transformer decoder). First, the deformable mixer encoder contains two types of operators: the channel-aware mixing operator leveraged to allow communication among different channels (i.e., efficient channel location mixing), and the spatial-aware deformable operator with deformable convolution applied to efficiently sample more informative spatial locations (i.e., deformed features). Second, the task-aware transformer decoder consists of the task interaction block and task query block. The former is applied to capture task interaction features via self-attention. The latter leverages the deformed features and task-interacted features to generate the corresponding task-specific feature through a query-based Transformer for corresponding task predictions. Extensive experiments on two dense image prediction datasets, NYUD-v2 and PASCAL-Context, demonstrate that our model uses fewer GFLOPs and significantly outperforms current Transformer-and CNN-based competitive models on a variety of metrics. The code are available at https://github.com/yangyangxu0/DeMT.

show abstract

“…[13] proposed a framework that tackles several language tasks but a single vision one. Differently, MulT [5] introduced a multitask transformer to handle multiple vision tasks. More complex vision transformer architectures have demonstrated that they outperform Convolutional Neural Network (CNN) based multitasking methods.…”

Section: Multitask Learning With Vision Transformersmentioning

confidence: 99%

“…This, in turn, allows us to achieve dense prediction in the comics domain by leveraging supervision from real-world annotations. Note that DTA differs from the co-attention introduced in prior works [5,6], wherein both cases, the attention is computed based on a specific task. By contrast, we learn a domain transferable attention between different domains.…”

Section: Domain Transferable Attentionmentioning

confidence: 99%

Dense Multitask Learning to Reconfigure Comics

Bhattacharjee

Süsstrunk

Salzmann

2023

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

In this paper, we develop a MultiTask Learning (MTL) model to achieve dense predictions for comics panels to, in turn, facilitate the transfer of comics from one publication channel to another by assisting authors in the task of reconfiguring their narratives. Our MTL method can successfully identify the semantic units as well as the embedded notion of 3D in comics panels. This is a significantly challenging problem because comics comprise disparate artistic styles, illustrations, layouts, and object scales that depend on the author's creative process. Typically, dense image-based prediction techniques require a large corpus of data. Finding an automated solution for dense prediction in the comics domain, therefore, becomes more difficult with the lack of ground-truth dense annotations for the comics images. To address these challenges, we develop the following solutions-we leverage a commonlyused strategy known as unsupervised image-to-image translation, which allows us to utilize a large corpus of realworld annotations; -we utilize the results of the translations to develop our multitasking approach that is based on a vision transformer backbone and a domain transferable attention module; -we study the feasibility of integrating our MTL dense-prediction method with an existing retargeting method, thereby reconfiguring comics.

show abstract

MuIT: An End-to-End Multitask Learning Transformer

Cited by 45 publications

References 78 publications

Inverted Pyramid Multi-task Transformer for Dense Scene Understanding

Inverted Pyramid Multi-task Transformer for Dense Scene Understanding

DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction

Dense Multitask Learning to Reconfigure Comics

Contact Info

Product

Resources

About