Building astroBERT, a language model for Astronomy &amp; Astrophysics

Grezes, Felix; Blanco-Cuaresma, S.; Accomazzi, Alberto; Kurtz, Michael J.; Shapurian, Golnaz; Henneken, Edwin A.; Grant, C. S.; Thompson, Donna M.; Chyla, Roman; McDonald, S.; Hostetler, Timothy W.; Templeton, Matthew R.; Lockhart, Kelly E.; Martinović, N.; Chen, Shinyi; Tanner, Chris C.; Protopapas, Pavlos

doi:10.48550/arxiv.2112.00590

Cited by 3 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One approach to generating scientific KGs is extracting entities from scientific documents [14,16,33]. There are multiple named-entity recognition (NER) algorithms to extract scientific entities, such as As-troBERT [13], and SciBERT [4]. These scientific entities can serve as the nodes of the KGs.…”

Section: Scientific Information Processingmentioning

confidence: 99%

“…is not specifically designed to recognize terms related to the NASA SMD. In the future, we aim to create NER algorithms similar to AstroBERT [13], and SciBERT [4]. In line with these two NER algorithms, we want to fine-tune models from the BERT family [9].…”

Section: Generating Nodesmentioning

confidence: 99%

“…We used the wikipedia module of Python to parse summaries from Wikipedia. Link to the wikipedia module: https://pypi.org/project/wikipedia/13 For k-means, we use the scikit-learn, also known as sklearn, a module of Python. Link to scikit-learn module: https://scikit-learn.org/…”

mentioning

confidence: 99%

See 2 more Smart Citations

NASA Science Mission Directorate Knowledge Graph Discovery

Timmer¹,

Khoo²,

Mark³

et al. 2023

Preprint

View full text Add to dashboard Cite

The size of the National Aeronautics and Space Administration (NASA) Science Mission Directorate (SMD) is growing exponentially, allowing researchers to make discoveries. However, making discoveries is challenging and time-consuming due to the size of the data catalogs, and as many concepts and data are indirectly connected. This paper proposes a pipeline to generate knowledge graphs (KGs) representing different NASA SMD domains. These KGs can be used as the basis for dataset search engines, saving researchers time and supporting them in finding new connections. We collected textual data and used several modern natural language processing (NLP) methods to create the nodes and the edges of the KGs. We explore the cross-domain connections, discuss our challenges, and provide future directions to inspire researchers working on similar challenges. CCS CONCEPTS• Information systems → Content analysis and feature selection; Language models.

show abstract

Section: Scientific Information Processingmentioning

confidence: 99%

Section: Generating Nodesmentioning

confidence: 99%

See 1 more Smart Citation

NASA Science Mission Directorate Knowledge Graph Discovery

Timmer¹,

Khoo²,

Mark³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…ADS already has a long tradition of openness, and with the expansion into SciX we will redouble our efforts to share our efforts with the larger research community. In 2021 we created and released a custom language model built on the astronomical literature called astroBERT (Grezes et al 2021). We have contributed data sets for the 2022 and 2023 data challenges at the first and second Workshops on Information Extraction from Scholarly Papers 3 .…”

Section: Future Ads: 2021-2030mentioning

confidence: 99%

Expansion of the NASA Astrophysics Data System to Earth and Space Sciences

Accomazzi¹

2023

Preprint

View full text Add to dashboard Cite

<p>The NASA Astrophysics Data System (ADS) is the primary Digital Library portal for Space Science Researchers. In addition to the scientific literature, the ADS has for a long time included in its database non-traditional scholarly resources such as research proposals, software packages, and high-level data products, making them discoverable and easily citable. Over the next three years, in response to NASA's efforts supporting interdisciplinary research and Open Science initiatives, the ADS will greatly expand its coverage of the literature, and will develop a new portal unifying access to the fields of Astrophysics, Planetary Science, Heliophysics, and Earth Science. It will also cover NASA funded research in Biological and Physical Sciences. The planned system will combine a scalable, discipline-agnostic core with a set of discipline specific knowledge centers which will curate and enrich its content using deep subject matter expertise from the NASA Science divisions. In this talk I will provide an overview of the ADS system, its distinguishing features, and then focus on our efforts to support and promote the FAIR principles as part of NASA's Year of Open Science initiatives.</p>

show abstract

“…Employing Google's Bidirectional Encoder Representations from Transformers (BERT; Devlin et al 2018) deep neural network architecture, Grezes et al (2021) have developed a domain-specific model for astronomy, termed astroBERT, through the training on a corpus comprising 395,499 astronomical research papers. Subsequently, this model was used for the development of the NER tool in ADS, which includes identifying specific organizations, projects, terms, etc., in the literature.…”

Section: Introductionmentioning

confidence: 99%

Astronomical Knowledge Entity Extraction in Astrophysics Journal Articles via Large Language Models

Shao,

Zhang,

et al. 2024

Res. Astron. Astrophys.

View full text Add to dashboard Cite

Astronomical knowledge entities, such as celestial object identifiers, are crucial for literature retrieval and knowledge graph construction, and other research and applications in the field of astronomy. Traditional methods of extracting knowledge entities from texts face numerous challenging obstacles that are difficult to overcome. Consequently, there is a pressing need for improved methods to efficiently extract them. This study explores the potential of pre-trained Large Language Models (LLMs) to perform astronomical knowledge entity extraction (KEE) task from astrophysical journal articles using prompts. We propose a prompting strategy called Prompt-KEE, which includes five prompt elements, and design eight combination prompts based on them. We select four representative LLMs (Llama-2-70B, GPT-3.5, GPT-4, and Claude 2) and attempt to extract the most typical astronomical knowledge entities, celestial object identifiers and telescope names, from astronomical journal articles using these eight combination prompts. To accommodate their token limitations, we construct two datasets: the full texts and paragraph collections of 30 articles. Leveraging the eight prompts, we test on full texts with GPT-4 and Claude 2, on paragraph collections with all LLMs. The experimental results demonstrate that pre-trained LLMs show significant potential in performing KEE tasks, but their performance varies on the two datasets. Furthermore, we analyze some important factors that influence the performance of LLMs in entity extraction and provide insights for future KEE tasks in astrophysical articles using LLMs. Finally, compared to other methods of knowledge entity extraction, LLMs exhibit strong competitiveness in multiple aspects.

show abstract

Building astroBERT, a language model for Astronomy & Astrophysics

Cited by 3 publications

References 3 publications

NASA Science Mission Directorate Knowledge Graph Discovery

NASA Science Mission Directorate Knowledge Graph Discovery

Expansion of the NASA Astrophysics Data System to Earth and Space Sciences

Astronomical Knowledge Entity Extraction in Astrophysics Journal Articles via Large Language Models

Contact Info

Product

Resources

About