Capreolus: A Toolkit for End-to-End Neural Ad Hoc Retrieval

Yates, Andrew; Arora, Siddhant; Zhang, Xinyu; Yang, Wei; Jose, Kevin Martin; Lin, Jimmy

doi:10.1145/3336191.3371868

Cited by 17 publications

(13 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, such supervision signals are often insufficient in many ranking scenarios. The less availability of relevance supervision pushes some Neu-IR methods to freeze their embeddings to avoid overfitting (Yates et al, 2020). The powerful deep pre-trained language models, such as BERT (Devlin et al, 2019), also do not effectively alleviate the dependence of Neu-IR on a large scale of relevance training signals.…”

Section: Related Workmentioning

confidence: 99%

Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

Sun¹,

Qian²,

Liu³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

The effectiveness of Neural Information Retrieval (Neu-IR) often depends on a large scale of in-domain relevance training signals, which are not always available in real-world ranking scenarios. To democratize the benefits of Neu-IR, this paper presents MetaAdaptRank, a domain adaptive learning method that generalizes Neu-IR models from label-rich source domains to few-shot target domains. Drawing on source-domain massive relevance supervision, MetaAdaptRank contrastively synthesizes a large number of weak supervision signals for target domains and meta-learns to reweight these synthetic "weak" data based on their benefits to the target-domain ranking accuracy of Neu-IR models. Experiments on three TREC benchmarks in the web, news, and biomedical domains show that MetaAdaptRank significantly improves the few-shot ranking accuracy of Neu-IR models. Further analyses indicate that MetaAdaptRank thrives from both its contrastive weak data synthesis and metareweighted data selection. The code and data of this paper can be obtained from https: //github.com/thunlp/MetaAdaptRank.

show abstract

Section: Related Workmentioning

confidence: 99%

Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

Sun¹,

Qian²,

Liu³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…MatchZoo focuses on TensorFlow and PyTorch (MatchZoo-py) implementations of neural reranking models, with experimental considerations like cross-validation and earlier parts of the pipeline left up to the user. OpenNIR handles both and defines a rigid pipeline similar to the "search-then-rerank" pipeline implemented by Capreolus v0.1 [29]. This pipeline orchestrates Ranker, Dataset, Trainer and Predictor modules to obtain first-stage retrieval results and then to rerank them.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, several toolkits have been proposed that implement reranking pipelines in the context of neural models [12,29]. In this setting, an efficient first-stage ranking method like BM25 uses an inverted index to identify a pool of candidate documents, and a neural model then reranks these candidate documents to form a final ranked list.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Flexible IR Pipelines with Capreolus

Yates

Jose

Zhang

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

While a number of recent open-source toolkits for training and using neural information retrieval models have greatly simplified experiments with neural reranking methods, they essentially hard code a "search-then-rerank" experimental pipeline. These pipelines consist of an efficient first-stage ranking method, like BM25, followed by a neural reranking method. Deviations from this setup often require hacks; some improvements, like adding a second reranking step that uses a more expensive neural method, are infeasible without major code changes. In order to improve the flexibility of such toolkits, we propose implementing experimental pipelines as dependency graphs of functional "IR primitives, " which we call modules, that can be used and combined as needed. For example, a neural IR pipeline may rerank results from a Searcher module that efficiently retrieves results from an Index module that it depends on. In turn, the Index depends on a Collection to index, which is provided by the pipeline. This Searcher module is self-contained: the pipeline does not need to know about or interact with the Index of the Searcher, which is transparently shared among Searcher modules when possible (e.g., a BM25 and a QL Searcher might share the same Index). Similarly, a Reranker module might depend on a Trainer (e.g., Tensorflow), feature Extractor, Tokenizer, etc. In both cases, the pipeline needs to interact only with the Reranker or Searcher directly; the complexity of their dependencies is hidden and intelligently managed. We rewrite the Capreolus toolkit to take this approach and demonstrate its use.

show abstract

“…In IR, there is a trend towards reproducibility, such as OpenNIR [9] and Capreolus [11] for Neural IR, or Anserini [10] for standard IR models. OpenNIR is based on its own configuration system (parameter files), Capreolus is based on Sacred, and Anserini is controlled by command line arguments.…”

Section: Figure 2: Datamaestro Text-related Repositorymentioning

confidence: 99%

Experimaestro and Datamaestro

Piwowarski¹

2020

Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Ensuring reproducibility is key to all scientific domains. As Information Retrieval (IR) experiments are often composed of several steps that can be shared between tested models, and rely on various resources, it is difficult to keep track of all the experimental settings and to ensure experiments can be reproduced easily. In this demo paper, we present two managers, Experimaestro and Datamaestro, and their add-ons for IR, designed to help to define and run experimental plans.

show abstract

Capreolus: A Toolkit for End-to-End Neural Ad Hoc Retrieval

Cited by 17 publications

References 11 publications

Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

Flexible IR Pipelines with Capreolus

Experimaestro and Datamaestro

Contact Info

Product

Resources

About