Examining influential factors for acknowledgements classification using supervised learning

Abstract

Acknowledgements have been examined as important elements in measuring the contributions to and intellectual debts of a scientific publication. Unlike previous studies that were limited in the scope of analysis and manual examination. The present study aimed to conduct the automatic classification of acknowledgements on a large scale of data. To this end, we first created a training dataset for acknowledgements classification by sampling the acknowledgements sections from the entire PubMed Central database. Second, we adopted various supervised learning algorithms to examine which algorithm performed best in what condition. In addition, we observed the factors affecting classification performance. We investigated the effects of the following three main aspects: classification algorithms, categories, and text representations. The CNN+Doc2Vec algorithm achieved the highest performance of 93.58% accuracy in the original dataset and 87.93% in the converted dataset. The experimental results indicated that the characteristics of categories and sentence patterns influenced the performance of classification. Most of the classifiers performed better on the categories of financial, peer interactive communication, and technical support compared to other classes.

Assessment of transparency indicators across the biomedical literature: How open is open?

“…Currently available tools can identify certain indicators of transparency, but they cannot be used to map and monitor these indicators across the published biomedical literature, their code is not openly available, their true performance is unknown, or they are paid services [ 23 – 26 ]. A recent publication utilized methods of machine learning to identify, among others, conflicts of interest and funding disclosures, but it can only process acknowledgments [ 27 ]. Another recently published tool, called SciScore, uses a machine learning method known as conditional random fields to identify measures of rigor (e.g., randomization, blinding, and power analysis) across the open access literature on PubMed and create a score of rigor and transparency.…”

Acknowledgement Entity Recognition in CORD-19 Papers

“…For example, Paul-Hus et al (2020) uses the Stanford NER module in NLTK to extract persons. Song et al (2020) also directly use people and organizations recognized by the Stanford CoreNLP (Manning et al, 2014). These works achieved a high recall by recognizing most name entities in the acknowledgements but ignored their relations to the papers where they appear, resulting in a fraction of entities that are mentioned but not actually acknowledged.…”

“…These works achieved a high recall by recognizing most name entities in the acknowledgements but ignored their relations to the papers where they appear, resulting in a fraction of entities that are mentioned but not actually acknowledged. Song et al (2020) consider grammar structure such as verb tense and voice and sentence patterns when labeling sentences to their six categories. For example, "was funded" is followed by an "organization".…”

“…Acknowledgements can be classified depending on the nature of the contribution. Song et al (2020) classified sentences in acknowledgement sections into 6 categories: declaration, financial, peer interactive communication and technical support, presentation, general acknowledgement, and general statement. They can also be classified based on the type of entities such as individual, organization, and grant.…”

Assessment of transparency indicators across the biomedical literature: how open is open?

“…Currently available tools can identify certain indicators of transparency, but they cannot be used to map and monitor these indicators across the published biomedical literature, their code is not openly available, their true performance is unknown or they are paid services [23–26]. A recent publication utilized methods of machine learning to identify, amongst others, conflicts of interest and funding disclosures, but it can only process acknowledgements [27]. Our work aims to expand our assessment of multiple indicators of transparency across the entire open biomedical literature, by developing and using tools and datasets that we hereby make freely available.…”