OmniNet: A unified architecture for multi-modal multi-task learning

Abstract

Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture which can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet. For example, a single instance of OmniNet can concurrently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning an unseen task. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet. 2

Multi-Task Learning with Deep Neural Networks: A Survey

“…This setup of task supervision is similar to the cascaded information architectures discussed in section 2.2.3. However, instead of hand-designing a hierarchy of tasks, this method performs a Figure 12: OmniNet architecture proposed in (Pramanik et al, 2019). Each modality has a separate network to handle inputs, and the aggregated outputs are processed by an encoder-decoder called the Central Neural Processor.…”

“…Both (Nguyen and Okatani, 2019;Akhtar et al, 2019) are focused on a set of tasks which all share the same fixed set of modalities. Instead, (Kaiser et al, 2017) and (Pramanik et al, 2019) focus on building a "universal multi-modal multi-task model", in which a single model can handle multiple tasks with varying input domains. The architecture introduced in (Kaiser et al, 2017) is comprised of an input encoder, an I/O mixer, and an autoregressive decoder.…”

“…The authors also demonstrate that the large degree of sharing between tasks yields significantly increased performance for tasks with limited training data. Instead of aggregating mechanisms from various modes of deep learning, (Pramanik et al, 2019) introduces an architecture called OmniNet with a spatio-temporal cache mechanism to learn dependencies across spatial dimensions of data as well as the temporal dimension. A diagram is shown in figure 12.…”

Exceeding the Limits of Visual-Linguistic Multi-Task Learning

“…However, such methods share a common goal of training a unified model over a group of tasks that performs well and limits requirements for task-specific parameters. Multi-task learning approaches have since been applied to numerous domains, such as forming sentence embeddings [46,51], solving computer vision tasks [26], and even performing multi-modal reasoning [37,39,41]. Several, more comprehensive, summaries of developments in the multi-task learning space are also available [45,59].…”

CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning

“…The dodecaDialogue task (Shuster et al, 2019) proposes twelve dialogue tasks, among which there are two language/vision tasks in which the agent has to generate a response for a given context. Other works try to exploit multi-task learning to improve on single-task model performance in discriminative tasks (Pramanik et al, 2019;Lu et al, 2019). Unfortunately, implementing multi-task learning using different datasets results is cumbersome (Subramanian et al, 2018).…”