Assessing publications through a bibliometric indicator: The case of comprehensive evaluation of scientific units in Poland

Kulczycki, Emanuel

doi:10.1093/reseval/rvw023

Cited by 60 publications

(58 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We observed somewhat different trends in the publication patterns in Flanders and Norway and could relate them to different designs. Quite different effects from those in Flanders and Norway have recently been reported from Poland (Kulczycki, 2017), which has the same type of PRFS with direct use of bibliometrics, but with a design quite different from those of the two other countries. The Stern review regards peer review as the cornerstone of research evaluation.…”

Section: Indicator-based Funding: the Norwegian Modelmentioning

confidence: 81%

Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective

Sivertsen

2017

Palgrave Commun

View full text Add to dashboard Cite

Inspired by The Metric Tide report (2015) on the role of metrics in research assessment and management, and Lord Nicholas Stern's report Building on Success and Learning from Experience (2016), which deals with criticisms of REF2014 and gives advice for a redesign of REF2021, this article discusses the possible implications for other countries. It also contributes to the discussion of the future of the REF by taking an international perspective. The article offers a framework for understanding differences in the motivations and designs of performance-based research funding systems (PRFS) across countries. It also shows that a basis for mutual learning among countries is more needed than a formulation of best practice, thereby both contributing to and correcting the international outlook in The Metric Tide report and its supplementary Literature Review.

show abstract

Section: Indicator-based Funding: the Norwegian Modelmentioning

confidence: 81%

Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective

Sivertsen

2017

Palgrave Commun

View full text Add to dashboard Cite

show abstract

“…Two solutions to this problem are to use publishing journal JIFs (or journal rankings: Kulczycki, 2017) as a proxy for citation impact or to use web-based early impact indicators. JIFs can avoid citing article publication delays if it is accepted that the average impact of a journal is an appropriate proxy for the impact of its articles (but see : Lozano, Larivière, & Gingras, 2012; and note also the time dimension : Larivière, Archambault, & Gingras, 2008) and that JIFs are stable over time (which is usually true: Thelwall & Fairclough, 2015).…”

Section: Introductionmentioning

confidence: 99%

Are Mendeley reader counts high enough for research evaluations when articles are published?

Thelwall

2017

AJIM

View full text Add to dashboard Cite

Design/methodology/approach -Average Mendeley reader counts were compared to average Scopus citation counts for 104520 articles from ten disciplines during the second half of 2016. Findings -Articles attracted, on average, between 0.1 and 0.8 Mendeley readers per article in the month in which they first appeared in Scopus. This is about ten times more than the average Scopus citation count. Research limitations/implications -Other subjects may use Mendeley more or less than the ten investigated here. The results are dependent on Scopus's indexing practices, and Mendeley reader counts can be manipulated and have national and seniority biases. Practical implications -Mendeley reader counts during the month of publication are more powerful than Scopus citations for comparing the average impacts of groups of documents but are not high enough to differentiate between the impacts of typical individual articles. Originality/value -This is the first multi-disciplinary and systematic analysis of Mendeley reader counts from the publication month of an article. IntroductionAcademic research is evaluated for appointment, promotion, tenure, for university league tables, for national research evaluation exercises and for self-reflection purposes. Some of these use quantitative data or are supported by numerical evidence of impact. Citation counts to refereed journal articles are a common source of this quantitative data, including in the form of Journal Impact Factors (JIFs) and field normalised citation counts (Garfield, 2006;Waltman, van Eck, van Leeuwen, Visser, & van Raan, 2011;Wilsdon, et al., 2015). Citation counts are not suitable for helping to evaluate new research because articles may take three years to attract a substantial number of citations due to publication delays. For this reason, formal evaluations often use a citation window of considerable length, such as three years (Wang, 2013), which excludes newer articles from evaluations. This means that the most recent and, therefore, most relevant research cannot be evaluated with the help of most citation-based indicators because they cannot differentiate effectively between different levels of impact for individual articles.Two solutions to this problem are to use publishing journal JIFs (or journal rankings: Kulczycki, 2017) as a proxy for citation impact or to use web-based early impact indicators. JIFs can avoid citing article publication delays if it is accepted that the average impact of a journal is an appropriate proxy for the impact of its articles (but see: Lozano, Larivière, & Gingras, 2012; and note also the time dimension: Larivière, Archambault, & Gingras, 2008) 1 Thelwall, M. (in press). Are Mendeley reader counts high enough for research evaluations when articles are published? Aslib Journal of Information Management, 69(2).

show abstract

“…The evaluation system gives 6 × 1 = 6. The value of k 1,4 indicates that two of the six papers should be in the third quartile, and so the biggest mark obtained with the evaluation system should be 4 × 1 + 2 × 2 = 8.…”

Section: (A)mentioning

confidence: 99%

“…The system provides a mark equal to 10 for their papers. Then the probabilistic interpretation of k 1,4 gives that the lowest mark should be greater than or equal to 3 × 2 + 2 × 1 = 8. Summing up the comments on the evaluation of both candidates, we obtain that the greatest possible mark for Candidate 1 equals the smallest possible mark for Candidate 2, and so the comparison of the confidence intervals gives that the result can be accepted, since the intersection of both of them is empty.…”

Section: (A)mentioning

confidence: 99%

“…An example of such issues is the procedure used by the Polish office of institutional research and assessment, whose performance-based research funding system is the so-called Comprehensive Evaluation of Scientific Units (CESU). In this case, the evaluation ranking is elaborated by a mixed procedure taking into account the bibliometric indices and the experts' assessment (see [1,2]). The discussion of which is the best way to define a rank-based evaluation system is one of the most recurrent topics in information science and its applications to research management.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Measuring Time-Dynamics and Time-Stability of Journal Rankings in Mathematics and Physics by Means of Fractional p-Variations

2017

View full text Add to dashboard Cite

Journal rankings of specific research fields are often used for evaluation purposes, both of authors and institutions. These rankings can be defined by means of several methods, as expert assessment, scholarly-based agreements, or by the ordering induced by a numeric index associated to the prestige of the journals. In order to be efficient and accepted by the research community, it must preserve the ordering over time, at least up to a point. Otherwise, the procedure for defining the ranking must be revised to assure that it reflects the presumably stable characteristic "prestige" that it claims to be quantifying. A mathematical model based on fractional p-variations of the values of the order number of each journal in a time series of journal rankings is explained, and its main properties are shown. As an example, we study the evolution of two given ordered lists of journals through an eleven-year series. These journal ranks are defined by using the 2-year Impact Factor of Thomson-Reuters (nowadays Clarivate Analytics) lists for MATHEMATICS and PHYSICS, APPLIED from 2002 to 2013. As an application of our model, we define an index that precludes the use of journal ranks for evaluation purposes when some minimal requirements on the associated fractional p-variations are not satisfied. The final conclusion is that the list of mathematics does not satisfy the requirements on the p-variations, while the list of applied physics does.

show abstract

Assessing publications through a bibliometric indicator: The case of comprehensive evaluation of scientific units in Poland

Cited by 60 publications

References 19 publications

Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective

Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective

Are Mendeley reader counts high enough for research evaluations when articles are published?

Measuring Time-Dynamics and Time-Stability of Journal Rankings in Mathematics and Physics by Means of Fractional p-Variations

Contact Info

Product

Resources

About