CompositeTasking: Understanding Images by Spatial Composition of Tasks

Abstract

We define the concept of CompositeTasking as the fusion of multiple, spatially distributed tasks, for various aspects of image understanding. Learning to perform spatially distributed tasks is motivated by the frequent availability of only sparse labels across tasks, and the desire for a compact multi-tasking network. To facilitate CompositeTasking, we introduce a novel task conditioning model -a single encoder-decoder network that performs multiple, spatially varying tasks at once. The proposed network takes an image and a set of pixel-wise dense task requests as inputs, and performs the requested prediction task for each pixel. Moreover, we also learn the composition of tasks that needs to be performed according to some CompositeTasking rules, which includes the decision of where to apply which task. It not only offers us a compact network for multitasking, but also allows for task-editing. Another strength of the proposed method is demonstrated by only having to supply sparse supervision per task. The obtained results are on par with our baselines that use dense supervision and a multi-headed multi-tasking design. The source code will be made publicly available at www.github.com/ nikola3794/composite-tasking.

Spatially Multi-conditional Image Generation

“…The visual quality when using 50% sparse labels is close to when using dense labels. This is interesting and desirable, since in practical scenarios one often ends up having sparse labels [42].…”

“…On the other hand, the sparsity is either simply caused by the label definition (e.g. sky has no normal) or due to missing annotations [42]. It is important to note that some geometric aspects of images, such as an object's depth and orientation, can be introduced manually (e.g.…”

“…This allows users to geometrically control images (beyond the semantics based control) both in the the presence or absence of 3D models. It goes without saying that the geometric manipulation of any image can be carried out by arXiv:2203.13812v2 [cs.CV] 14 Jul 2022 first inferring its geometric attributes using existing methods [42,49,61], followed by generation after manipulation.…”