SciBERT: A Pretrained Language Model for Scientific Text

Beltagy, Iz; Lo, Kyle; Cohan, Arman

doi:10.48550/arxiv.1903.10676

Cited by 244 publications

(347 citation statements)

References 0 publications

Supporting

Mentioning

341

Contrasting

Order By: Relevance

“…We also fine-tune these models with text from our target datasets using the standard language model training objective to evaluate the effects of exposing task specific content. Additionally, we compare with SciBERT [40], a language model based on BERT but pre-trained on a large corpus of scientific text, to assess the impact of structure inclusion on different types of Transformer models. For comparisons with equivalent loss functions, we use SentBERT [13] that uses triples of anchor, semantically similar and dissimilar sentences to learn the embeddings.…”

Section: Evaluation Frameworkmentioning

confidence: 99%

Structure and Semantics Preserving Document Representations

Raman¹,

Shah²,

Veloso³

2022

Preprint

View full text Add to dashboard Cite

Retrieving relevant documents from a corpus is typically based on the semantic similarity between the document content and query text. The inclusion of structural relationship between documents can benefit the retrieval mechanism by addressing semantic gaps. However, incorporating these relationships requires tractable mechanisms that balance structure with semantics and take advantage of the prevalent pre-train/fine-tune paradigm. We propose here a holistic approach to learning document representations by integrating intra-document content with inter-document relations. Our deep metric learning solution analyzes the complex neighborhood structure in the relationship network to efficiently sample similar/dissimilar document pairs and defines a novel quintuplet loss function that simultaneously encourages document pairs that are semantically relevant to be closer and structurally unrelated to be far apart in the representation space. Furthermore, the separation margins between the documents are varied flexibly to encode the heterogeneity in relationship strengths. The model is fully fine-tunable and natively supports query projection during inference. We demonstrate that it outperforms competing methods on multiple datasets for document retrieval tasks.

show abstract

Section: Evaluation Frameworkmentioning

confidence: 99%

Structure and Semantics Preserving Document Representations

Raman¹,

Shah²,

Veloso³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For representing terms, we used a BERT-based model, called SciBERT [1] with 12 attention layers and the hidden layer of size 768 which was pretrained on 1.14M papers from Semantic Scholar 2 resulting in a corpus size of 3.17B tokens. SciBERT was then fine-tuned on the entire ACL corpus to represent semantic meanings of words in each time frame.…”

Section: B Text Representationmentioning

confidence: 99%

“…We propose a novel change-oriented summarization approach based on word semantic evolution to provide information on key changes that occurred in the data over time. As an underlying data source we use the Association for Computational Linguistics (ACL) dataset, and specifically, the Parcit Structured XML 1 version of the ACL Anthology Reference Corpus [2]. By analyzing this data we provide answers to questions about what concepts are highly changing and what are their directions of changes.…”

Section: Introductionmentioning

confidence: 99%

“…Examples can been seen in news [13,19,16], social networks [10], and scholarly articles [12]. Several strategies have been proposed to understand the significance of individual topics [18], to learn how multidisciplinary research area is founded [17], 1 https://acl-arc.comp.nus.edu.sg/archives/acl-arc-160301-parscit or to illustrate the evolution of scientific communities [4]. A common approach is by frequency analysis [7], or using topic modelling, so that the correlations between link citations or the sets of co-occurring words are uncovered [5,6].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Change Summarization of Diachronic Scholarly Paper Collections by Semantic Evolution Analysis

Paharia¹,

Pozi²,

Jatowt³

2021

Preprint

View full text Add to dashboard Cite

The amount of scholarly data has been increasing dramatically over the last years. For newcomers to a particular science domain (e.g., IR, physics, NLP) it is often difficult to spot larger trends and to position the latest research in the context of prior scientific achievements and breakthroughs. Similarly, researchers in the history of science are interested in tools that allow them to analyze and visualize changes in particular scientific domains. Temporal summarization and related methods should be then useful for making sense of large volumes of scientific discourse data aggregated over time. We demonstrate a novel approach to analyze the collections of research papers published over longer time periods to provide a high level overview of important semantic changes that occurred over the progress of time. Our approach is based on comparing word semantic representations over time and aims to support users in better understanding of large domain-focused archives of scholarly publications. As an example dataset we use the ACL Anthology Reference Corpus that spans from 1979 to 2015 and contains 22,878 scholarly articles.

show abstract

“…The sub-tokens can then spuriously match other terms, including sub-tokens from other split identifiers, but not the whole, unsplit term. Researchers in the biomedical domain recognized the same issue affecting medical terms and proposed domain-specific BERT adaptations [2,11,26].…”

Section: #Bugsmentioning

confidence: 99%

Fast Changeset-based Bug Localization with BERT

Ciborowska¹,

Damevski²

2021

Preprint

View full text Add to dashboard Cite

Automatically localizing software bugs to the changesets that induced them has the potential to improve software developer efficiency and to positively affect software quality. To facilitate this automation, a bug report has to be effectively matched with source code changes, even when a significant lexical gap exists between natural language used to describe the bug and identifier naming practices used by developers. To bridge this gap, we need techniques that are able to capture software engineering-specific and project-specific semantics in order to detect relatedness between the two types of documents that goes beyond exact term matching. Popular transformer-based deep learning architectures, such as BERT, excel at leveraging contextual information, hence appear to be a suitable candidate for the task. However, BERT-like models are computationally expensive, which precludes them from being used in an environment where response time is important.In this paper, we describe how BERT can be made fast enough to be applicable to changeset-based bug localization. We also explore several design decisions in using BERT for this purpose, including how best to encode changesets and how to match bug reports to individual changes for improved accuracy. We compare the accuracy and performance of our model to a non-contextual baseline (i.e., vector space model) and BERT-based architectures previously used in software engineering. Our evaluation results demonstrate advantages in using the proposed BERT model compared to the baselines, especially for bug reports that lack any hints about related code elements.

show abstract

SciBERT: A Pretrained Language Model for Scientific Text

Cited by 244 publications

References 0 publications

Structure and Semantics Preserving Document Representations

Structure and Semantics Preserving Document Representations

Change Summarization of Diachronic Scholarly Paper Collections by Semantic Evolution Analysis

Fast Changeset-based Bug Localization with BERT

Contact Info

Product

Resources

About