Citations take time to accumulate and citations that do happen are not all the same, some are simply reporting results and mentioning prior art, while others are independent replications or criticism. Citation patterns over time are key to understanding and characterizing research outputs and bibliometrics since they reveal the influence a publication has over time. However, studies on citation patterns over time have tended to focus on simple citation counts that don't provide nuance to the types of citations being made. In the below post, we take a look at the number of days until citations occur broken out by the type of citation, whether it is supporting, mentioning, or contrasting the paper being cited, and where the citation occurs in the citing paper i.e. introduction, methods, results, or discussion.

What is interesting about the analysis below is that the average time to receive a supporting or contrasting citation differs noticeably from citations in general in that supporting and contrasting citations take longer time to accrue. Additionally, in the general distribution of citations over time, it appears that there is some peak where most publications receive their first citation after which being cited becomes less and less likely . However, this is not clearly seen in contrasting and supporting citations which appear to continue to grow. These early results indicate that critical discussion takes longer to accumulate but doesn't stop growing proportionally to the date of the publication.

As an impact measure, citations are often used to quantify how a publication has been received. However, simple citation counts are not granular enough to show what kind of citations, and thereby attention, a publication has received. scite provides a corpus of citations that surfaces the types of citations being made, whether they are supporting, mentioning, or contrasting citations and the sections of citations such as whether the citation comes from introduction, methods, results, or discussion sections (Nicholson et al., 2021).

By looking at these citations according to how they are talking about a publication or where they appear in the paper we are better able to get a sense of the critical reception of a publication. Has its claims been supported? Are they disputed? Were this papers methods influential? etc.

Analysis

In order to look at citation patterns broken out by citation type and originating section, we looked at the number of days until the first citation of each type and section. We looked at two datasets, the number of days until a citation occured to publications from 2019 and from 2018. In order to ensure research published earlier in the year wasn't given longer time to accrue citations, we looked at 1 year and 2 year windows from the time of publication. Table 1 and 2 describe a summary of those datasets. These summary statistics show the count of publications included and other standard statistics. What is particularly interesting is comparing the mean and median citation numbers.

Table 1: Days until first citation to publications in 2019 under a 1 year window

Table 2: Days until first citation to publications in 2018 under a 2 year window

In order to get a better picture of those tables, Figure 1 and Figure 2 show letter-value plots illustrating the distribution of days until first citation.

Figure 1: First occurrence of citation by citation type and section to publications in 2019 under a 1 year window

Figure 2: First occurrence of citation by citation type and section to publications in 2018 under a 2 year window

However, Figures 1 and 2 don't really show the different distribution shapes clearly when the same data is presented in a histogram: see Figure 3 and 4 below.

Figure 3: Distribution of first citation occurrence by citation type and section to publications in 2019 under a 1 year window

Figure 4: Distribution of first citation occurrence by citation type and section to publications in 2018 under a 2 year window

Discussion

The primary characteristics of interest are how the distribution and average time to first citation vary across citation type and originating section.

The most noticeable of these is the differences in mean and median time to a first supporting or contrasting citation. For the 2 year window to publications in 2018, the mean time to accumulate a citation was 335 days, *yet the mean time to accumulate a supporting citation was 419 days and contrasting was 448 days which is a 113 day difference. In the longer time period, we can also see that citations from methods, results, and discussion sections do take slightly longer to accrue.

The findings above indicate that a citation supporting a publication takes longer to occur than other types of citations. This is intuitive since a reproduction of results takes longer than mentioning a publication's results. Drawing on methods or discussing results from a paper can also be done right away. Generally, this indicates that there is quantitative support for supporting citations to take more effort than other types of citations. Interestingly, contrasting citations take even longer to accrue! Indicating that critical citations take even more effort than reproduction of results. Perhaps part of this is that robust criticism involves careful reproduction of results sometimes. It may also be that a critical reading and analysis of a publication takes time to formulate and work into future studies.

Another noticeable trend is that the distribution of first citations tends to peak, this is clearly seen in the 2 year window. Again, supporting and contrasting citations appear to be the exception. Interestingly methods and results sections also appear to be an exception. Where for supporting, contrasting, methods, and results sections there doesn't appear to be a peak and accumulation of a first citation tends to grow. The above results should be taken with caution since in Figure 3, patterns that might have appeared to continue growing show a peak (Figure 4) over longer periods. In future studies longer time windows should be looked at to see if this pattern holds

The growing distribution for supporting, contrasting, methods, and results citations is interesting and may indicate that critical discussion isn't subject to the same citation patterns as others. A few things this could indicate are as follows. Researchers are often looking for novel methods and results and are willing to cite older and uncited works to utilize novel methods and results in their works. Another point worth exploring that could explain the above distribution for supporting and contrasting citations is that as papers get older they become easier to criticise or support. This could be because the methods and theoretical foundations they used had more time to receive support or criticism itself or that it becomes cheaper and easier to replicate experiments over time. Also, perhaps many citations are subject to a popularity effect, where recency determines the likelihood of getting cited uncritically. Maybe in this case researchers are more likely to critically cite or support a work that hasn't been cited yet since time has passed where popular ways of citing the paper are no longer as relevant indicating a lexico-semantic shift in how the citations are written. Finally, there may be an aspect where contrasting, supporting, methods, and results citations are more novel than other types of citations and over time the likelihood of a new idea of how to use or talk about a publication increases.

Generally however, these characteristics appear to be inline with the cumulative effect of citations over time as discussed elsewhere (Glänzel, Schlemmer, and Thijs, 2004). In future studies we should look to see how the analysis done above differs with longer time scales and more sophisticated analysis such as using survival analysis as seen in (Lee et al., 2021). By looking at more sophisticated approaches over longer time scales we can explore some of the ideas proposed above such as whether critical citations take more effort or whether growing citations to supporting, contrasting, methods, and results sections can be tied to increasing novelty in how the citations are made.

References

Glänzel, W., Schlemmer, B., & Thijs, B. (2004). Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon. Scientometrics, 58, 571-586.

Lee, S., Schneijderberg, C., Kim, Y., & Steinhardt, I. (2021). Have Academics' Citation Patterns Changed in Response to the Rise of World University Rankings? A Test Using First-Citation Speeds. Sustainability.

Nicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N., Grabitz P., & Rife, S. (2021). scite: a smart citation index that displays the context of citations and classifies their intent using deep learning. bioRxiv.