Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity

Abstract

We present a new scientific document similarity model based on matching fine-grained aspects of texts. To train our model, we exploit a naturally-occurring source of supervision: sentences in the full-text of papers that cite multiple papers together (co-citations). Such cocitations not only reflect close paper relatedness, but also provide textual descriptions of how the co-cited papers are related. This novel form of textual supervision is used for learning to match aspects across papers. We develop multi-vector representations where vectors correspond to sentence-level aspects of documents, and present two methods for aspect matching: (1) A fast method that only matches single aspects, and (2) a method that makes sparse multiple matches with an Optimal Transport mechanism that computes an Earth Mover's Distance between aspects. Our approach improves performance on document similarity tasks in four datasets. Further, our fast single-match method achieves competitive results, paving the way for applying finegrained similarity to large scientific corpora. 1

Impact of supervisors' research style on young biomedical scientists' capacity development as measured by REDi, a novel index of crossdisciplinarity

“…The most basic idea to calculate REDi is the fact that papers cite strongly related papers to their own research. There is a study by Mysore et al ( 2022 ) that is based on the same idea. In this study, the degree of relatedness strength is replaced by the distance between the fields.…”

SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

“…In the scientific domain, contrastive learning of cross-document links (e.g. citations) has led to improved document-level representations (Cohan et al, 2020;Ostendorff et al, 2022b;Mysore et al, 2022). These representations can be indexed and consumed later by lightweight downstream models without additional fine-tuning.…”

“…Further, we use this benchmark to investigate and improve the generalization ability of document representation models. Following recent work (Cohan et al, 2020;Ostendorff et al, 2022b;Mysore et al, 2022) we pre-fine-tune a transformer model originally trained on citation triplets to produce high-quality representations for downstream tasks. We hypothesize that condensing all relevant information of a document into a single vector might not be expressive enough for generalizing across a wide range of tasks.…”

“…Prior work addresses a similar challenge in the context of document similarity and learns multiple representations associated with different aspects of a paper (e.g. task, method, results) (Mysore et al, 2022;Ostendorff et al, 2022a). In contrast, we aim to learn effective representations for multiple downstream task formats.…”

“…We attempt to learn task-specific embeddings of documents by pre-fine-tuning on multiple objectives simultaneously. Ostendorff et al (2022a) and Mysore et al (2022) study the orthogonal task of generating multiple embeddings per paper for different "facets," while we aim to learn general embeddings for mutliple task formats.…”