Assessing the quality of scientific conferences based on bibliographic citations

Abstract

Assessing the quality of scientific conferences is an important and useful service that can be provided by digital libraries and similar systems. This is specially true for fields such as Computer Science and Electric Engineering, where conference publications are crucial. However, the majority of the existing quality metrics, particularly those relying on bibliographic citations, has been proposed for measuring the quality of journals. In this article we conduct a study about the relative performance of existing journal metrics in assessing the quality of scientific conferences. More importantly, departing from a deep analysis of the deficiencies of these metrics, we propose a new set of quality metrics especially designed to capture intrinsic and important aspects related to conferences, such as longevity, popularity, prestige, and periodicity. To demonstrate the effectiveness of the proposed metrics, we have conducted two sets of experiments that contrast their results against a ''gold standard'' produced by a large group of specialists. Our metrics obtained gains of more than 12% when compared to the most consistent journal quality metric and up to 58% when compared to standard metrics such as Thomson's Impact Factor.

Relationship between high-quality journals and conferences in computer vision

“…This is similar to previous results (Goodrum et al 2001) in computer science (but using CiteSeer data). There has been other bibliometric research on conference papers in computer science as well (Martins et al 2010;Franceschet 2011).…”

Software engineering research in Brazil from the perspective of young researchers: a panorama of the last decade

“…Analyzing the distribution of published papers by the selected young researchers according to QUALIS levels 8 (see Table 9), we observe a high concentration of papers in the top strata of the QUALIS system (from level A1 to B1), particularly for journals (58 %). A smaller percentage of papers published in top strata conferences (32 %) may be due to constant participation on local workshops and theme-focused conferences, which do not usually attain high levels in the QUALIS system despite of their importance to build local research communities and develop international relationships, respectively.…”

“…Martins et al [8] have evaluated the usefulness of journal-oriented publication quality metrics to address the importance of research conferences. They propose variants of the well-known impact factor metrics to consider the importance of longevity, size, periodicity, and prestige of conferences.…”

“…6 DBLP Computer Science Bibliography: http://www.informatik.uni-trier.de/~ley/db/ 7 The QUALIS system, managed by CAPES, classifies the most important computer science conferences and journals according to a ranking system consisting of the 7 levels: A1 (higher), A2, B1, B2, B3, B4, and B5 (lower). 8 Only papers published in venues classified by QUALIS system were considered in this analysis.…”

How healthy are software engineering conferences?

“…We have used SHINE (the Simple H-INdex Estimator [23]) to rank the conferences and show, e.g., that higher-impact conferences attract more submissions but tend to have lower acceptance rates. Numerous alternative approaches to ranking scientific venues have been proposed (e.g., [45,[54][55][56][57][58][59][60][61]), but they fall beyond the scope of this paper. For example, da Silva et al [59] propose a ranking scheme for scientific conferences based on ranking the PC members.…”