ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

Abstract

Learning visual representations from natural language supervision has recently shown great promise in a number of pioneering works. In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets/tasks. However, it remains a challenge to evaluate the transferablity of these foundation models due to the lack of easy-to-use toolkits for fair benchmarking. To tackle this, we build ELEVATER 1 , the first benchmark to compare and evaluate pre-trained language-augmented visual models. Several highlights include: (i) Datasets. As downstream evaluation suites, it consists of 20 image classification datasets and 35 object detection datasets, each of which is augmented with external knowledge. (ii) Toolkit. An automatic hyper-parameter tuning toolkit is developed to ensure the fairness in model adaption. To leverage the full power of language-augmented visual models, novel language-aware initialization methods are proposed to significantly improve the adaption performance. (iii) Metrics. A variety of evaluation metrics are used, including sample-efficiency (zero-shot and few-shot) and parameter-efficiency (linear probing and full model fine-tuning). We will release our toolkit and evaluation platforms for the research community.

OmDet: Large‐scale vision‐language multi‐dataset pre‐training with multimodal detection network

“…We focus on three types of downstream tasks for evaluation: Object Detection in the Wild: object detection in the wild test a model's ability to adapt to various different domains with drastically different label sets. ELEVATER [24] is a new object detection benchmark that is composed of 35 diverse real-world challenging domains with full-shot, few-shot and zero-shot training settings. Note that there are two variations of data used in prior work [20].…”

“…The proposed method is evaluated on three downstream tasks: object detection in the wild (ODinW) [24], openvocabulary detection, and phrase grounding [25]. Results show that OmDet is able to outperform all prior art, including the powerful GLIP [20] that is pre-trained on much larger datasets.…”

Parameter-Efficient Model Adaptation for Vision Transformers

“…For few-shot benchmark experiments, we conduct experiments on 20 image classification datasets from the EL-EVATER benchmark (Li et al 2022b) on four Quadro RTX A6000 GPUs. Detailed dataset statistics are given in the supplementary material.…”

“…For benchmark experiments, we use the SGD (Ruder 2016) optimizer with the learning rate and weight decay being automatically searched for all methods so that these two hyperparameters have the optimum combination. We borrow the automatic hyper-parameter tuning toolkit from Li et al (2022b). Training epochs are set via grid search.…”

“…The averaged 5-shot experimental result on Full-model Fine-tuning, Linear-probing, Adapter-tuning, LoRA, Compacter, and KAdaptation (ours) across 20 datasets from ELEVATER benchmark(Li et al 2022b) in terms of accuracy (%) and number of trainable parameters (in millions) (#Params) across random seeds of {0, 1, 2}. The ViT-B-224/32 via CLIP pretraining is evaluated.…”