Perceiver IO: A General Architecture for Structured Inputs & Outputs

Abstract

The recently-proposed Perceiver model obtains good results on several domains (images, audio, multimodal, point clouds) while scaling linearly in compute and memory with the input size. While the Perceiver supports many kinds of inputs, it can only produce very simple outputs such as class scores. Perceiver IO overcomes this limitation without sacrificing the original's appealing properties by learning to flexibly query the model's latent space to produce outputs of arbitrary size and semantics. Perceiver IO still decouples model depth from data size and still scales linearly with data size, but now with respect to both input and output sizes. The full Perceiver IO model achieves strong results on tasks with highly structured output spaces, such as natural language and visual understanding, StarCraft II, and multi-task and multi-modal domains. As highlights, Perceiver IO matches a Transformer-based BERT baseline on the GLUE language benchmark without the need for input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation. Code: https://dpmd.ai/perceiver-code Preprint. Under review.

Multiview Transformers for Video Recognition

“…Note that the only parameters introduced into the model from this fusion method are the linear projections of bottleneck tokens from one view to the next, and the bottleneck tokens themselves which are learned from random initialization. We also note that "bottleneck" tokens have also been used by [31,49].…”

Input-level Inductive Biases for 3D Reconstruction

“…It was noted a decade ago that big data along with simple architectures are "unreasonably effective" [13] at solving many perception problems, and subsequent progress has only reinforced this [50]. Computer vision has moved from architectures like ConvNets, which are highly general image processors [26], to methods that are based on Transformers [54] such as ViT [9] and Perceivers [20,21], where the underlying Transformer can be equally effective across multiple domains like sound and language. Unifying architectures is useful because architectural improvements can be propagated across tasks and domains trivially.…”

“…1), like a loadable software solution instead of a more rigid hardware solution. As the general model we employ the recently published Perceiver IO [20] and as domain we focus on multiview geometry and 3D reconstruction, an area of computer vision where architectural specialization is particularly exuberant [17,19,27,33,41,62,65].…”

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval

“…In this case random modality dropout could be used during training as e.g. done in AVSlowfast [41] or Perceiver [21].…”

“…Although modality agnostic transformers such as Perceive-rIO [21] have been proposed, they have been constructed with a different goal of learning a latent space that can cover multiple tasks in different domains. Compared to our work, the latent space in such cases mainly serves the purpose of compressing multiple inputs and tasks in one model.…”