Specialized document embeddings for aspect-based similarity of research papers

Abstract

Document embeddings and similarity measures underpin contentbased recommender systems, whereby a document is commonly represented as a single generic embedding. However, similarity computed on single vector representations provides only one perspective on document similarity that ignores which aspects make two documents alike. To address this limitation, aspect-based similarity measures have been developed using document segmentation or pairwise multi-class document classification. While segmentation harms the document coherence, the pairwise classification approach scales poorly to large scale corpora. In this paper, we treat aspect-based similarity as a classical vector similarity problem in aspect-specific embedding spaces. We represent a document not as a single generic embedding but as multiple specialized embeddings. Our approach avoids document segmentation and scales linearly w.r.t. the corpus size. In an empirical study, we use the Papers with Code corpus containing 157, 606 research papers and consider the task, method, and dataset of the respective research papers as their aspects. We compare and analyze three generic document embeddings, six specialized document embeddings and a pairwise classification baseline in the context of research paper recommendations. As generic document embeddings, we consider FastText, SciBERT, and SPECTER. To compute the specialized document embeddings, we compare three alternative methods inspired by retrofitting, fine-tuning, and Siamese networks. In our experiments, Siamese SciBERT achieved the highest scores. Additional analyses indicate an implicit bias of the generic document embeddings towards the dataset aspect and against the method aspect of each research paper. Our approach of aspect-based document embeddings mitigates potential risks arising from implicit biases by making them explicit. This can, for example, be used for more diverse and explainable recommendations.

Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection

“…The shallow notion of what composes paraphrases used by these systems limits their understanding of the task and makes it challenging to interpret detection decisions in practice. For example, although high structural and grammatical similarities can indicate plagiarism, detection systems are often not concerned with the aspects that make two texts segments alike (Ostendorff et al, 2022;Wahle et al, 2022a).…”

Application of a multimodal model optimized by multi-head-attention mechanism in the classification of fishery resource remote sensing data files

“…For example, Konaray SK et al used some experiments to analyze file extension features and used machine learning algorithms to match file suffixes and three-byte magic header information of files to achieve accurate classification of file types [1].Josh Angichiodo et al proposed the use of generative adversarial networks (SGAN) to identify file contents by means of the problem of classifying hidden files and files after their extensions or headers have been obfuscated, this adversarial network performs supervised learning based on the contents to achieve the file classification work [2]. The second one is the clustering of document content and its similarity.Malte Ostendorff,Till Blume et al have proposed Specialized Document Embeddings for Aspect-based Similarity of Research Papers [3], which represents the content of a document as a generic embedding. It treats the similarity of documents in a given aspect as a vector similarity problem in a specific embedding space.…”

“…Content-based document classification is implemented using a specialized document embedding design. In addition to these classification studies for physical documents, there are many classification methods that focus only on data content, for example, Ren F et al vectorized text content using Word2Vec word embedding method or Bert model method, and then classified text content using a bidirectional long short-term memory recurrent neural network (BiLSTM) [3]. Kang Chen and Huazheng Fu analyzed the lexical features of short paths and transformed the path word embedding features into feature images using recurrent neural networks (RNN) on the dataset [4].…”

SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

“…Prior work addresses a similar challenge in the context of document similarity and learns multiple representations associated with different aspects of a paper (e.g. task, method, results) (Mysore et al, 2022;Ostendorff et al, 2022a). In contrast, we aim to learn effective representations for multiple downstream task formats.…”

“…We attempt to learn task-specific embeddings of documents by pre-fine-tuning on multiple objectives simultaneously. Ostendorff et al (2022a) and Mysore et al (2022) study the orthogonal task of generating multiple embeddings per paper for different "facets," while we aim to learn general embeddings for mutliple task formats.…”