Investigating disagreement in the scientific literature

Lamers, Wout S.; Boyack, Kevin W.; Larivière, Vincent; Sugimoto, Cassidy R.; Waltman, Ludo; Murray, Dakota

doi:10.7554/elife.72737

Cited by 31 publications

(22 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another category of problems concerns whether citations really capture the idea of "influence" or impact: not all influences are cited and some works are cited that have no influence (so-called perfunctory citations). There are different types of citations (Bornmann and Daniel, 2008a), which do not show an equal influence, with some citations for example being negative (Lamers et al, 2021). This does not necessarily mean that highly-cited publications are not influential.…”

Section: B Metricsmentioning

confidence: 99%

Citation models and research evaluation

Traag¹

2022

Preprint

View full text Add to dashboard Cite

Citations in science are being studied from several perspectives. On the one hand, there are approaches such as scientometrics and the science of science, which take a more quantitative perspective. In this chapter I briefly review some of the literature on citations, citation distributions and models of citations. These citations feature prominently in another part of the literature which is dealing with research evaluation and the role of metrics and indicators in that process. Here I briefly review part of the discussion in research evaluation. This also touches on the subject of how citations relate to peer review. Finally, I try to integrate the two literatures with the aim of clarifying what I believe the two can learn from each other. The fundamental problem in research evaluation is that research quality is unobservable. This has consequences for conclusions that we can draw from quantitative studies of citations and citation models. The term "indicators" is a relevant concept in this context, which I try to clarify. Causality is important for properly understanding indicators, especially when indicators are used in practice: when we act on indicators, we enter causal territory. Even when an indicator might have been valid, through its very use, the consequences of its use may invalidate it. By combining citation models with proper causal reasoning and acknowledging the fundamental problem about unobservable research quality, we may hope to make progress.

show abstract

Section: B Metricsmentioning

confidence: 99%

Citation models and research evaluation

Traag¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We study the dual question: whether negative papers (in our sense) receive more citations. Lamers et al [33] study disagreement in science across diverse fields, which is a related concept to that of negative citations, finding that there is highest disagreement in the social sciences and humanities, and lowest disagreement in mathematics and computer science (of which AI is a subfield).…”

Section: Related Workmentioning

confidence: 99%

Did AI get more negative recently?

et al. 2023

View full text Add to dashboard Cite

In this paper, we classify scientific articles in the domain of natural language processing (NLP) and machine learning (ML), as core subfields of artificial intelligence (AI), into whether (i) they extend the current state-of-the-art by the introduction of novel techniques which beat existing models or whether (ii) they mainly criticize the existing state-of-the-art, i.e. that it is deficient with respect to some property (e.g. wrong evaluation, wrong datasets, misleading task specification). We refer to contributions under (i) as having a ‘positive stance’ and contributions under (ii) as having a ‘negative stance’ (to related work). We annotate over 1.5 k papers from NLP and ML to train a SciBERT-based model to automatically predict the stance of a paper based on its title and abstract. We then analyse large-scale trends on over 41 k papers from the last approximately 35 years in NLP and ML, finding that papers have become substantially more positive over time, but negative papers also got more negative and we observe considerably more negative papers in recent years. Negative papers are also more influential in terms of citations they receive.

show abstract

“…In addition, we note the importance for scientific reasoning of displaying disputes and controversies. A recent bibliometric analysis, studying scientific disagreements on a large scale [12], highlights the importance of "relative differences" as well as the "heterogeneity" of observed practices of disagreements. We therefore speculate that the representation of differences in retrieval practices of predecessors in cliques, which is a result that AI can perform with neutrality, can help users verify and secure their own assessment shifts on methods or on results, with regard to the observed usage of the other members of a cognitive community, labeled by keywords.…”

Section: Modeling An 'Assessor's Shift' In Assistance To Scientific R...mentioning

confidence: 99%

A Multiverse Graph to Help Scientific Reasoning from Web Usage: Interpretable Patterns of Assessor Shifts in GRAPHYP

et al. 2023

View full text Add to dashboard Cite

The digital support for scientific reasoning presents contrasting results. Bibliometric services are improving, but not academic assessment; no service for scholars relies on logs of web usage to base query strategies for relevance judgments (or assessor shifts). Our Scientific Knowledge Graph GRAPHYP innovates with interpretable patterns of web usage, providing scientific reasoning with conceptual fingerprints and helping identify eligible hypotheses. In a previous article, we showed how usage log data, in the form of ‘documentary tracks’, help determine distinct cognitive communities (called adversarial cliques) within sub-graphs. A typology of these documentary tracks through a triplet of measurements from logs (intensity, variety and attention) describes the potential approaches to a (research) question. GRAPHYP assists interpretation as a classifier, with possibilistic graphical modeling. This paper (Paper 2) shows what this approach can bring to scientific reasoning; it involves visualizing complete interpretable pathways, in a multi-hop assessor shift, which users can then explore toward the ‘best possible solution’—the one that is most consistent with their hypotheses. Applying the Leibnizian paradigm of scientific reasoning, GRAPHYP highlights infinitesimal learning pathways, as a ‘multiverse’ geometric graph in modeling possible search strategies answering research questions.

show abstract

Investigating disagreement in the scientific literature

Cited by 31 publications

References 58 publications

Citation models and research evaluation

Citation models and research evaluation

Did AI get more negative recently?

A Multiverse Graph to Help Scientific Reasoning from Web Usage: Interpretable Patterns of Assessor Shifts in GRAPHYP

Contact Info

Product

Resources

About