Flexible retrieval with NMSLIB and FlexNeuART

Boytsov, Leonid; Nyberg, Eric

doi:10.18653/v1/2020.nlposs-1.6

Cited by 16 publications

(21 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Long MS MARCO doc documents are truncated to 445 first BERT tokens, but such shortening leads to only small (≈ 1%) loss in accuracy [4]. Experiments are carried out using a retrieval toolkit FlexNeuART [5]. We measure effectiveness using the mean reciprocal rank (MRR), which is an official metric for MS MARCO data [7].…”

Section: Methodsmentioning

confidence: 99%

A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models

Mokrii

Boytsov

Braslavski

2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

Due to high annotation costs, making the best use of existing human-created training data is an important research direction. We, therefore, carry out a systematic evaluation of transferability of BERT-based neural ranking models across five English datasets. Previous studies focused primarily on zero-shot and few-shot transfer from a large dataset to a dataset with a small number of queries. In contrast, each of our collections has a substantial number of queries, which enables a full-shot evaluation mode and improves reliability of our results. Furthermore, since source datasets licences often prohibit commercial use, we compare transfer learning to training on pseudo-labels generated by a BM25 scorer. We find that training on pseudo-labels-possibly with subsequent fine-tuning using a modest number of annotated queries-can produce a competitive or better model compared to transfer learning. However, there is a need to improve the stability and/or effectiveness of the few-shot training, which, in some cases, can degrade performance of a pretrained model.

show abstract

Section: Methodsmentioning

confidence: 99%

A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models

Mokrii

Boytsov

Braslavski

2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

show abstract

“…Translation-based features Capturing semantic relationships between a query and a document is also crucial to improving retrieval accuracy. To incorporate such features, we can use a translation model (Boytsov and Nyberg, 2020;Boytsov and Kolter, 2021) to measure the log translation probability between queries and documents. The conditional probability we need p(q|d n ) is generated by the IBM Model 1 translation model, and the final query-document feature is the sum of all individual conditional query probabilities.…”

Section: Learning-to-rank Featuresmentioning

confidence: 99%

Learning to Rank in the Age of Muppets: Effectiveness–Efficiency Tradeoffs in Multi-Stage Ranking

Zhang¹,

Hu²,

Liu³

et al. 2021

Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing

View full text Add to dashboard Cite

It is well known that rerankers built on pretrained transformer models such as BERT have dramatically improved retrieval effectiveness in many tasks. However, these gains have come at substantial costs in terms of efficiency, as noted by many researchers. In this work, we show that it is possible to retain the benefits of transformer-based rerankers in a multi-stage reranking pipeline by first using feature-based learning-to-rank techniques to reduce the number of candidate documents under consideration without adversely affecting their quality in terms of recall. Applied to the MS MARCO passage and document ranking tasks, we are able to achieve the same level of effectiveness, but with up to 18× increase in efficiency. Furthermore, our techniques are orthogonal to other methods focused on accelerating transformer inference, and thus can be combined for even greater efficiency gains. A higher-level message from our work is that, even though pretrained transformers dominate the modern IR landscape, there are still important roles for "traditional" LTR techniques, and that we should not forget history.1 Muppets being a whimsical way to refer to BERT and related transformer models.

show abstract

“…These algorithms are also widely used in the industry at scale. However, all known graph indices are static and do not support updates, especially delete requests [18], possibly due to the fact that simple graph modification rules for insertions and deletions do not retain the same graph quality over a stream of insertions and deletions.…”

Section: Shortcoming Of Existing Algorithmsmentioning

confidence: 99%

“…As a result, the current practice in industry is to periodically re-build such indices from scratch [18] to manifest recent changes to the underlying dataset. However, this is a very expensive operation.…”

Section: Shortcoming Of Existing Algorithmsmentioning

confidence: 99%

“…While graph-indices offer state-of-the-art search performance, all known algorithms apply for the static-ANNS problem. In particular, deletions pose a big challenge for all these algorithms -e.g., see this discussion [18] on HNSW supporting delete requests by adding them to a blacklist and omitting from search results. Arguably, this is due to the lack of methods which modify the navigable graphs while retaining the original search quality.…”

Section: Why Are Deletions Hard?mentioning

confidence: 99%

See 1 more Smart Citation

FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search

Singh,

Subramanya,

Krishnaswamy

et al. 2021

Preprint

View full text Add to dashboard Cite

Approximate nearest neighbor search (ANNS) is a fundamental building block in information retrieval with graphbased indices being the current state-of-the-art [7] and widely used in the industry. Recent advances [51] in graph-based indices have made it possible to index and search billion-point datasets with high recall and millisecond-level latency on a single commodity machine with an SSD.However, existing graph algorithms for ANNS support only static indices that cannot reflect real-time changes to the corpus required by many key real-world scenarios (e.g. index of sentences in documents, email or a news index). To overcome this drawback, the current industry practice for manifesting updates into such indices is to periodically re-build these indices, which can be prohibitively expensive.In this paper, we present the first graph-based ANNS index that reflects corpus updates into the index in real-time without compromising on search performance. Using update rules for this index, we design FreshDiskANN, a system that can index over a billion points on a workstation with an SSD and limited memory, and support thousands of concurrent real-time inserts, deletes and searches per second each, while retaining > 95% 5-recall@5. This represents a 5-10x reduction in the cost of maintaining freshness in indices when compared to existing methods.

show abstract

Flexible retrieval with NMSLIB and FlexNeuART

Cited by 16 publications

References 52 publications

A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models

A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models

Learning to Rank in the Age of Muppets: Effectiveness–Efficiency Tradeoffs in Multi-Stage Ranking

FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search

Contact Info

Product

Resources

About