Overview of ChEMU 2020: Named Entity Recognition and Event Extraction of Chemical Reactions from Patents

He, Jiayuan; Nguyen, Dat Quoc; Akhondi, Saber A.; Druckenbrodt, Christian; Thorne, Camilo; Hoessel, Ralph; Afzal, Zubair; Zhai, Zenan; Fang, Biaoyan; Yoshikawa, Hiyori; Albahem, Ameer; Cavedon, Lawrence; Cohn, Trevor; Baldwin, Timothy; Verspoor, Karin

doi:10.1007/978-3-030-58219-7_18

Cited by 27 publications

(35 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ChEMU 2020 benchmark dataset 1 [He et al, 2021] contains snippets sampled from 170 English patents from the European Patent Office and the United States Patent and Trademark Office [He et al, 2020a[He et al, ,b, 2021. As shown in Fig.…”

Section: Benchmark Data For Chemical Entity Recognition -Chemu 2020 Dmentioning

confidence: 99%

“…Particularly, we leverage different pre-trained models available in the literature to create ensembles of named entity recognizers. We evaluate our models in chemistry, clinical and wet lab corpora provided in the context of the ChEMU (Cheminformatics Elsevier Melbourne University) [He et al, 2020a], DEFT (Défi Fouille de Textes) [Grabar et al, 2018] and WNUT (Workshop on Noisy User-generated Text) [Tabassum et al, 2020] challenges, respectively. Our results show that ensembles of named entity recognizers based on masked language models can achieve effective NER performances in these different domains and languages.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Ensemble of deep masked language models for effective named entity recognition in multi-domain corpora

Naderi

Knafou

Copara

et al. 2021

Preprint

View full text Add to dashboard Cite

The health and life science domains are well-known for their wealth of entities. These entities are presented as free text in large corpora, such as biomedical scientific and electronic health records. To enable the secondary use of these corpora and unlock their value, named entity recognition (NER) methods are proposed. Inspired by the success of deep masked language models, we present an ensemble approach for NER using these models. Results show statistically significant improvement of the ensemble models over baselines based on individual models in multiple domains - chemical, clinical and wet lab - and languages - English and French. The ensemble model achieves an overall performance of 79.2% macro F1-score, a 4.6 percentage point increase upon the baseline in multiple domains and languages. These results suggests that ensembles are a more effective strategy for tackling NER. We further perform a detailed analysis of their performance based on a set of entity properties.

show abstract

Section: Benchmark Data For Chemical Entity Recognition -Chemu 2020 Dmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Ensemble of deep masked language models for effective named entity recognition in multi-domain corpora

Naderi

Knafou

Copara

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In this paper, we provide a detailed overview of the activities within ChEMU2020 lab, including the new ChEMU corpus, the tasks, the evaluation framework, the evaluation results, and a summary of participants' approaches. This paper is an extension of our previous overview papers (He et al, 2020a , b ) and thereby the task descriptions (section 4) and core evaluation results (section 5) are repeated here from those papers. Our focus is to provide additional detail about the preparation of the corpus we developed (section 3) and to provide more comprehensive analysis of the evaluation results.…”

Section: Introductionmentioning

confidence: 97%

“…The ChEMU (Cheminformatics Elsevier Melbourne University) lab is an initiative to encourage research on methods for automated information extraction from chemical patents. As a first running of ChEMU, ChEMU2020 lab focused on extraction of chemical reactions from patents (He et al, 2020a ; Nguyen et al, 2020 ). We prepared two fundamental information extraction tasks.…”

Section: Introductionmentioning

confidence: 99%

ChEMU 2020: Natural Language Processing Methods Are Effective for Information Extraction From Chemical Patents

Nguyen

Akhondi

et al. 2021

Front. Res. Metr. Anal.

Self Cite

View full text Add to dashboard Cite

Chemical patents represent a valuable source of information about new chemical compounds, which is critical to the drug discovery process. Automated information extraction over chemical patents is, however, a challenging task due to the large volume of existing patents and the complex linguistic properties of chemical patents. The Cheminformatics Elsevier Melbourne University (ChEMU) evaluation lab 2020, part of the Conference and Labs of the Evaluation Forum 2020 (CLEF2020), was introduced to support the development of advanced text mining techniques for chemical patents. The ChEMU 2020 lab proposed two fundamental information extraction tasks focusing on chemical reaction processes described in chemical patents: (1) chemical named entity recognition, requiring identification of essential chemical entities and their roles in chemical reactions, as well as reaction conditions; and (2) event extraction, which aims at identification of event steps relating the entities involved in chemical reactions. The ChEMU 2020 lab received 37 team registrations and 46 runs. Overall, the performance of submissions for these tasks exceeded our expectations, with the top systems outperforming strong baselines. We further show the methods to be robust to variations in sampling of the test data. We provide a detailed overview of the ChEMU 2020 corpus and its annotation, showing that inter-annotator agreement is very strong. We also present the methods adopted by participants, provide a detailed analysis of their performance, and carefully consider the potential impact of data leakage on interpretation of the results. The ChEMU 2020 Lab has shown the viability of automated methods to support information extraction of key information in chemical patents.

show abstract

“…To reduce the time and effort needed for information extraction from chemical literature and patents, datasets and text mining approaches have been developed on a wide range of information extraction tasks, including named entity recognition and relation extraction [7,8,9,10,11,12]. Most of these methods focus on processing plain text by leveraging state of the art Natural Language Processing (NLP) approaches, and tabular data is usually ignored or discarded, which causes significant loss in the amount of compound-related information that can be extracted from patents.…”

Section: Introductionmentioning

confidence: 99%

ChemTables: A Dataset for Semantic Classification of Tables in Chemical Patents

Zhai

Druckenbrodt

Thorne

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Chemical patents are a commonly used channel for disclosing novel compounds and reactions, and hence represent important resources for chemical and pharmaceutical research. Key chemical data in patents is often presented in tables. Both the number and the size of tables can be very large in patent documents. In addition, various types of information can be presented in tables in patents, including spectroscopic and physical data, or pharmacological use and effects of chemicals. Since images of Markush structures and merged cells are commonly used in these tables, their structure also shows substantial variation. This heterogeneity in content and structure of tables in chemical patents makes relevant information difficult to find. We therefore propose a new text mining task of automatically categorising tables in chemical patents based on their contents. Categorisation of tables based on the nature of their content can help to identify tables containing key information, improving the accessibility of information in patents that is highly relevant for new inventions. For developing and evaluating methods for the table classification task, we developed a new dataset, called ChemTables, which consists of 7,886 chemical patent tables with labels of their content type. We introduce this data set in detail. We further establish strong baselines for the table classification task in chemical patents by applying state-of-the-art neural network models developed for natural language processing, including TabNet, ResNet and Table-BERT on ChemTables. The best performing model, Table-BERT, achieves a performance of 88.66 micro F1 score on the table classification task. Availability: A 10% sample of the ChemTables dataset has been made publicly available, subject to a data usage agreement.

show abstract

Overview of ChEMU 2020: Named Entity Recognition and Event Extraction of Chemical Reactions from Patents

Cited by 27 publications

References 11 publications

Ensemble of deep masked language models for effective named entity recognition in multi-domain corpora

Ensemble of deep masked language models for effective named entity recognition in multi-domain corpora

ChEMU 2020: Natural Language Processing Methods Are Effective for Information Extraction From Chemical Patents

ChemTables: A Dataset for Semantic Classification of Tables in Chemical Patents

Contact Info

Product

Resources

About