Deep Cross Residual Learning for Multitask Visual Recognition

Abstract

Residual learning has recently surfaced as an effective means of constructing very deep neural networks for object recognition. However, current incarnations of residual networks do not allow for the modeling and integration of complex relations between closely coupled recognition tasks or across domains. Such problems are often encountered in multimedia applications involving large-scale content recognition. We propose a novel extension of residual learning for deep networks that enables intuitive learning across multiple related tasks using cross-connections called cross-residuals. These cross-residuals connections can be viewed as a form of innetwork regularization and enables greater network generalization. We show how cross-residual learning (CRL) can be integrated in multitask networks to jointly train and detect visual concepts across several tasks. We present a single multitask cross-residual network with >40% less parameters that is able to achieve competitive, or even better, detection performance on a visual sentiment concept detection problem normally requiring multiple specialized single-task networks. The resulting multitask cross-residual network also achieves better detection performance by about 10.4% over a standard multitask residual network without cross-residuals with even a small amount of cross-task weighting.

A Modulation Module for Multi-task Learning with Applications in Image Retrieval

“…Although this problem is rarely identified in the literature, many of the existing methods are in fact designed to mitigate destructive interference in multi-task learning. For example, in the popular multi-branch neural network architecture and its variants, the task-specific branches are designed carefully with the prior knowledge regarding the relationships of certain tasks [18,8,20]. By doing this, people expect less conflicting training signals to the shared parameters.…”

Learning Task Relatedness in Multi-Task Learning for Images in Context

“…Additionally, with MTL comes a natural urge to simplify the models at hand and group the tasks that would benefit each other's learning process. With this mutually beneficial task relationship in mind, there are numerous domains and modalities [10,14,15,24,37,39,47,51] where the MTL methodology can be applied. As such, MTL is often used implicitly without a specific reference in methods such as transfer learning and fine-tuning [4,40] as well.…”

Many Task Learning With Task Routing

“…In our work we address the feature robustness issue by randomizing the sharing structure from the start of the training process and enforcing tasks to use alternate routes for their data-flow through the model. Both symmetric and asymmetric MTL approaches often rely on prior knowledge to help with architecture design, sharing options and task grouping [23,32,10,1]. If such knowledge is present it is a helpful resource for designing an MTL model.…”