The Efficiency Misnomer

Dehghani, Mostafa; Arnab, Anurag; Beyer, Lucas; Vaswani, Ashish; Tay, Yi

doi:10.48550/arxiv.2110.12894

Cited by 16 publications

(22 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Limitations: There is no perfect metric for measuring the overall performance of a given neural network architecture [14]. We have provided 4 different metrics but there are probably some aspects that are not considered.…”

Section: Discussionmentioning

confidence: 99%

Augmenting Convolutional networks with attention-based aggregation

Touvron¹,

Cord²,

El-Nouby³

et al. 2021

Preprint

View full text Add to dashboard Cite

We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We replace the final average pooling by an attentionbased aggregation layer akin to a single transformer block, that weights how the patches are involved in the classification decision. We plug this learned aggregation layer with a simplistic patch-based convolutional network parametrized by 2 parameters (width and depth). In contrast with a pyramidal design, this architecture family maintains the input patch resolution across all the layers. It yields surprisingly competitive trade-offs between accuracy and complexity, in particular in terms of memory consumption, as shown by our experiments on various computer vision tasks: object classification, image segmentation and detection.

show abstract

Section: Discussionmentioning

confidence: 99%

Augmenting Convolutional networks with attention-based aggregation

Touvron¹,

Cord²,

El-Nouby³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Lastly, as one exception to general trends, the training latency of GAT on HPO-Metab is higher than that of SubGNN. Note that latency ignores the parallelism from large batch sizes (Dehghani et al, 2021). Our model can show relatively high latency since it requires full batch computation.…”

Section: Models For S2n Translated Graphsmentioning

confidence: 99%

“…We conduct experiments with four real-world datasets to evaluate the performance and efficiency of S2N translation. We measure the number of parameters, throughput (samples per second), and latency (seconds per forward pass) for efficiency (Dehghani et al, 2021). We demonstrate that models with S2N translation are more efficient than the existing approach without a significant performance drop.…”

Section: Introductionmentioning

confidence: 99%

Efficient Representation Learning of Subgraphs by Subgraph-To-Node Translation

Kim¹,

Oh²

2022

Preprint

View full text Add to dashboard Cite

A subgraph is a data structure that can represent various real-world problems. We propose Subgraph-To-Node (S2N) translation, which is a novel formulation to efficiently learn representations of subgraphs. Specifically, given a set of subgraphs in the global graph, we construct a new graph by coarsely transforming subgraphs into nodes. We perform subgraph-level tasks as node-level tasks through this translation. By doing so, we can significantly reduce the memory and computational costs in both training and inference. We conduct experiments on four real-world datasets to evaluate performance and efficiency. Our experiments demonstrate that models with S2N translation are more efficient than state-of-theart models without substantial performance decrease.

show abstract

“…This section provides a complementary study that compares ARTEMIS' complexity with the complexity of several approaches. Since no single measure is enough to assess the complexity of a model (Dehghani et al, 2021), we compare the different models on three metrics:…”

Section: Complexity and Efficiency Studymentioning

confidence: 99%

ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity

Delmas¹,

Rezende²,

Larlus³

2022

Preprint

View full text Add to dashboard Cite

An intuitive way to search for images is to use queries composed of an example image and a complementary text. While the first provides rich and implicit context for the search, the latter explicitly calls for new traits, or specifies how some elements of the example image should be changed to retrieve the desired target image. Current approaches typically combine the features of each of the two elements of the query into a single representation, which can then be compared to the ones of the potential target images. Our work aims at shedding new light on the task by looking at it through the prism of two familiar and related frameworks: text-to-image and image-to-image retrieval. Taking inspiration from them, we exploit the specific relation of each query element with the targeted image and derive light-weight attention mechanisms which enable to mediate between the two complementary modalities. We validate our approach on several retrieval benchmarks, querying with images and their associated free-form text modifiers. Our method obtains state-of-the-art results without resorting to side information, multi-level features, heavy pre-training nor large architectures as in previous works.

show abstract

The Efficiency Misnomer

Cited by 16 publications

References 49 publications

Augmenting Convolutional networks with attention-based aggregation

Augmenting Convolutional networks with attention-based aggregation

Efficient Representation Learning of Subgraphs by Subgraph-To-Node Translation

ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity

Contact Info

Product

Resources

About