Event-based Video Reconstruction Using Transformer

Weng, Wenming; Zhang, Yueyi; Xiong, Zhiwei

doi:10.1109/iccv48922.2021.00256

Cited by 70 publications

(39 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In encoder-decoder layers, the decoder also includes a CA sub-layer after SA, which allows it to attend to memory embeddings (sometimes referred to as context) provided by the encoder. In video, encoder-decoder architectures are exploited mostly for captioning [54], [55], [56], [57], [58], [59] and video generation [60], [61], [62]. These three configurations, however, are not the only possibility.…”

Section: Transformer Trends Adopted For Videomentioning

confidence: 99%

“…Alternatively, [126] borrows from NLP and utilizes LSTMs to embed local temporal information into the input. Works using a hybrid ConvLSTM [149] are also found [62], [123]. Finally, in some instances, networks pre-trained to perform an auxiliary task (regarded as experts) are used to pre-process the input and provide specific information that can be leveraged by the Transformer [66], [131].…”

Section: Embeddingmentioning

confidence: 99%

“…These three configurations, however, are not the only possibility. Multiple encoder-decoder pairs can be applied in parallel [62] or sequentially [63], encoders can run in parallel [50] or hierarchically [64], [65], or encoder+decoder can be combined in a single module [58], [66]. Special tokens.…”

Section: Transformer Trends Adopted For Videomentioning

confidence: 99%

“…As mentioned in Sec. 3.2.2, weight sharing is equivalent to batched processing or applying the same module to different clips in a video [71], spatiotemporal scales [62], and modalities [49]. In fact, most hierarchical designs share weights for the modules that process independent neighborhoods in the first stage [11], [12], [14], [80], [90], [99].…”

Section: Weight Sharingmentioning

confidence: 99%

“…Some train in a supervised fashion [9], [11], [48], [60], [90], [130], directly for a downstream task on large datasets, such as HowTo100M [148], Kinetics-700 [147], or even ImageNet21K [144]. Others follow a similar fashion but utilizing small datasets aided by some data augmentation [62], [88], [173]. There are also works leveraging selfsupervised losses [49], [51], [80], [114], [115] on medium to large datasets.…”

Section: Training Regimementioning

confidence: 99%

See 4 more Smart Citations

Video Transformers: A Survey

Selva¹,

Johansen²,

Escalera³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformer models have shown great success modeling long-range interactions. Nevertheless, they scale quadratically with input length and lack inductive biases. These limitations can be further exacerbated when dealing with the high dimensionality of video. Proper modeling of video, which can span from seconds to hours, requires handling long-range interactions. This makes Transformers a promising tool for solving video related tasks, but some adaptations are required. While there are previous works that study the advances of Transformers for vision tasks, there is none that focus on in-depth analysis of video-specific designs. In this survey we analyse and summarize the main contributions and trends for adapting Transformers to model video data. Specifically, we delve into how videos are embedded and tokenized, finding a very widspread use of large CNN backbones to reduce dimensionality and a predominance of patches and frames as tokens. Furthermore, we study how the Transformer layer has been tweaked to handle longer sequences, generally by reducing the number of tokens in single attention operation. Also, we analyse the self-supervised losses used to train Video Transformers, which to date are mostly constrained to contrastive approaches. Finally, we explore how other modalities are integrated with video and conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D CNN counterparts with equivalent FLOPs and no significant parameter increase.

show abstract