“I hope that the manuscript did not accidentally reference something retracted… or heavily contrasted by others.” - Everyone involved in scientific publishing

This article will cover:

• How you can use Reference Check to be more confident in manuscripts you submit or review
• How to use it on a sample PDF
• How to interpret the results
• Who it’s relevant for
• Some FAQ towards the end

It also assumes that you:

• Understand what Smart Citations are and how they are classified (See: How does scite work?)

Reference Check

We explored elsewhere how scite’s Smart Citations work and how they can help you more quickly evaluate a research paper.

Now let’s see how you can improve other aspects of your research process with scite.

This document is focused on Reference Check, a feature we have that improves the publishing and peer-review process by allowing you to quickly understand:

• how a manuscript uses its references
• how each each reference itself has been cited by others
• whether there is anything to worry about any of the references (e.g. editorial notices such as retractions, or if any have a lot of contrasting findings).

We will caveat that though Reference Check is a premium feature, it is possible to try it out with our one month free trial when you register a new account. For the purposes of this article, you can follow along using an example of the output here.

What exactly is Reference Check?

Reference Check is a tool that takes a PDF and generates a report that allows you to:

• See how the manuscript cites its own references
• See whether any have received editorial notice(s)
• Check how many times each reference has been supported or contrasted

Okay, simple enough. How do I use it?

Here are the steps to run a Reference Check:

• Log into scite
• Open the Product tab and click on Reference Check
• Upload a PDF of your manuscript
• Simply wait for the report generation to complete and it will take you to the report
• Your reference checks can be accessed from your Profile, under Reference Checks

Awesome. What does all of this mean?

If you haven’t generated your own Reference Check, you can follow along using this example, which was a Reference Check generated for a sample manuscript entitled “Eosinophils support adipocyte maturation and promote glucose tolerance in obesity”.

Immediately, we see that:

• scite detected 41 references from this paper
• One of those references has an editorial concern
• Combined, the detected references have received ~41K citations from other publications

At this point we might wonder:

• Which reference has an editorial notice?
• What type of editorial notice did it receive?
• How many times did the uploaded PDF cite this reference?
• Are those citations to the dubious reference justified, or are they something to worry about?

Let’s take a look further below to answer these questions.

Looking immediately below the title section, we see that there are filters on the left, and citation statements on the right.

Each of these citation statements show text extracted from the uploaded paper (also referred to as the source paper) and show where it makes a citation to each reference (also referred to as the target paper).

Here, we immediately see from the top-right that the citation statements are ordered by editorial concern, and that the first two citation statements cite the same target paper: “Cytokines suppress adipogenesis…” by Suzawa et al. (2003).

In addition to seeing that our manuscript makes two references to Suzawa (2003), we also see that this Suzawa (2003) reference has two editorial notices: a retraction and a correction.

In a matter of seconds, we were able to identify that our manuscript has two citations to a retracted paper. Even more importantly, we can see exactly how our manuscript used that reference by reading the extracted citation snippet, and use that to inform our decision about the quality of the manuscript itself.

Got it! So… who would use this?

While the Reference Check feature is valuable for anyone involved in publishing a manuscript, it is fundamentally a tool that allows you to see how a given paper uses its references and to evaluate the quality of those references.

An editor at a journal might be interested in using it to improve how quickly and reliably they can evaluate incoming manuscripts.

An author, or a co-author, of a manuscript might use it to triple check their piece before submitting it to a journal.

Anyone evaluating a full-text PDF of a publication might be curious to quickly understand how it makes use of its references, and gain insight into any potential concerns about them.

How exactly is this different from your scite Report Page?

For a given publication, its scite Report Page shows you all of the times it was cited by other works in the field. Phrased differently, in the Report Page, the publication of interest is the target while each other paper showing up in the results is the source of the citation.

On the contrary, for a given publication, its Reference Check output shows you how it cites other works in the field. Phrased differently, in the Reference Check, the publication of interest is the source while each other paper showing up in the results is the target of the citation.

For more information about the report page, please see this article: How do I use the scite report page?

How do you detect the references in a paper?

scite uses machine learning to automatically identify references from manuscripts and to match citation statements with their respective references. The ability to identify and match references depends upon the format of the references, whether or not the reference has a DOI, as well as the format of the PDF.

For example, citations to news articles or policy documents that typically do not have DOIs will not appear in the Reference Check report. Therefore, scite will nearly always miss a few references in our Reference Check.

What happens to the PDF I upload?

When you upload a PDF to generate a Reference Check report, the only thing we store and persist in our database is the information contained in the report. That is:

• The title and other metadata
• The references used
• The citation statements to each reference from the manuscript

In the process of generating the report, the uploaded PDF is stored in a private, restricted S3 bucket. The file is automatically deleted after the report is generated.

Do you have an API for this?

Yes! We have an API where you can programmatically upload PDF files to run reference checks. In order to learn more, please reach out to us at sales@scite.ai