People around the world spend billions of hours streaming the latest show or movie through services like Netflix, Hulu, or Disney +. Such binge-watching has nearly doubled during the pandemic, making deciding what to watch a nightly conversation for many households. The pandemic has also accelerated the dissemination of research, with journals speeding up their peer review processes and many scientists choosing to share their work ahead of formal publication on preprint servers–online repositories that allow researchers to share their work without peer review.

Crowd-sourced review and recommendation sites such as MetaCritic and Rotten Tomatoes have helped ease the information overload problem with tv shows and movies, but no such service really exists in research. Choosing the wrong movie or show can be a nuisance; following the wrong line of research can be deadly.

What if we adopted the same processes that exist to help us find the best TV show, movie, book, or album to science articles?

The challenge: how to effectively evaluate scientific literature

For Netflix and other streaming services, it is easy to see what critics have said about the show by checking out services such as Rotten Tomatoes that aggregate short reviews from journalists around the world and indicates if it is a favorable review or not, essentially giving viewers the capability of seeing dozens of newspaper reviews at once.

For scientific research, there is no such platform that allows audiences to easily determine what the scientific literature says about a research article. There are platforms that list which articles refer to an article of interest. However, this approach is similar to simply listing which newspapers mention a movie — it’s better than nothing, but it doesn’t make it easy to see what they say.

Other heuristics, such as the reputation of the journal, the authors, and the organization they are affiliated with, provide some information but not in enough detail.

New alternative metrics, such as altmetrics, show how many times scientific articles have been mentioned on social media and the news but this says nothing of quality. Scite does have a peer-review process, but because it is almost universally closed, readers can’t assess it and with preprints there is no peer review.

It is easier to determine how a television show or movie has been assessed by reviewers than it is for scientific papers.

scite: Rotten Tomatoes for Science

What if, like online movie review aggregators such as Rotten Tomatoes, researchers, policymakers, and anyone interested in science could see at a glance what all citing papers say about an article of interest without having to open every single citing paper?

This is exactly what we’ve built at scite: a system that allows anyone to see how an article has been cited, not just how many times. We do this through Smart Citations, which are citations that show an excerpt of text from the citing paper along with a description if that text supports or disputes the original article.

These smart citations allow readers to see if an article of interest has been cited by others, how it has been cited, and if it has been supported or disputed. Additionally, with up-to-date data on retractions, errata, corrections, and expressions of concern, users can easily surface if a paper has been corrected or retracted.

How scite Works in Practice

To see how this can help, it’s best to look at the world as it exists without scite and the world as it exists with scite. The preprint, “Presence of SARS-CoV-2 reactive T cells in COVID-19 patients and healthy donors” was posted without peer review on April 22nd on medRxiv, a medical preprint server hosted by BMJ, Yale, and Cold Spring Harbor.

The paper has since been picked up by 56 news outlets, 12 blogs, 3112 tweets, 5 Facebook pages, and one video. In short, it has received a massive amount of attention. But is it right? The measures used to assess scientific papers mentioned above all look solely at impact but a high citation count can occur on a flawed study, and a paper can be tweeted incessantly or amplified by the news without ever even being read. Reading the paper, or consulting with an expert, is preferable but not feasible given the volume of papers being published.

Looking at the same preprint through the lens of scite, you can quickly see that it has been cited 23 times by 13 different articles, 5 of which indicate that it has been supported. This information is available to readers anywhere they are reading an article online through the Chrome or Firefox browser extensions.

Clicking through, one can not only see that this article has been supported but how it has been supported by directly reading what the other scientific articles say about it.

You can also visualize this more explicitly through our very recently launched citation network visualizations.

This information shouldn’t just live on scite or in a browser extension but should exist on the version of record, the scientific article and anywhere the article is shared, like news sites and Wikipedia. Adoption of scite Smart Citation badges is happening with journals like Journal of Cell Biology, Journal of Experimental Medicine, and Journal of General Physiology displaying smart citations as well as other search platforms like Europe PMC embedding Smart Citations into their metrics.

Even more recently, the Washington Post has in some cases started linking to scite reports to help their readers better evaluate studies.

We’re encouraged that hundreds of more journals will soon do the same.

Next Steps for scite

Join scite in helping to fight information overload in science:

• Researchers: Sign up at scite to better discover and evaluate scientific papers more effectively
• Journalists: Check scite when fact checking or referencing a scientific paper.
• Publishers: Add the scite badge to your site (it’s free) and reach out to us about making your content discoverable on scite.

Learn more at scite.ai.