Can’t bibliometric analysts do better? How quality assessment without field expertise does not work

Abstract

The article is an invited comment on Guy Madison and Therese Söderlund (M&S): Comparisons of content and scientific quality indicators across peer-reviewed journal articles with more or less gender perspective: Gender studies can do better. Scientometrics 115(3):1161–1183. The article pinpoints a series of serious problems in M&S’s quantitative quality assessment and analysis of the field of gender studies, pertaining to their overall conceptual framework, their general approach and their specific analysis. It is argued that the over-arching problem in M&S’s study is their lack of expert knowledge of the field of gender studies, their lack of respect for differences between qualitative and quantitative research, and their research design, which is biased towards quantitative social and natural science research. Firstly, it is demonstrated that a key concept, ‘gender perspective’, is used in an incoherent and confusing way in M&S’s analysis. Secondly, it is argued that the confusion is not an isolated definitional problem, but related to a series of slippages between M&S’s source of inspiration (Ganetz in Genusvetenskapliga projektansökningar inom humaniora-samhällsvetenskap – en uppföljning av Vetenskapsrådets beredning och utfall år 2004. Vetenskapsrådets rapportserie, Stockholm 15/2005, 2005) and their own adoption of the category. Thirdly, differences between qualitative and quantitative research, and between hermeneutic and explanatory knowledge production, are discussed more broadly to sustain the argument that the mentioned slippages occur, because M&S transfer analytical tools from Ganetz’ qualitative study, based on a peer review methodology, to a quantitative quality assessment, carried out without field specific expert knowledge. It is argued that, to be adequate and relevant, a quality assessment would need to respect these differences, and develop tools and research designs accordingly. Fourthly, the validity of M&S’s content analysis—the core of their study—is questioned in detail because of its use of inadequate analytical categories, and because of its exclusion of central elements from the analysis. Finally, it is argued that the bias in M&S’s research design is reproduced in their results.

The Long March Through the Patriarchal Institutions: A Dialogue Between Rosi Braidotti and Nina Lykke

“…I have recently, together with colleagues in Sweden, replied to an attack on Swedish gender research by two researchers from the University of Umeå (Lykke, 2018). These researchers have in a pseudo-scientifically-objective way argued that the Swedish Research Council has given far too many resources to gender studies in recent decades (Madison and Söderlund 2018).…”

«Estudos sobre as mulheres» em Portugal Pós-Declaração de Pequim – Estudo bibliométrico das revistas ex æquo e Faces de Eva

“…Essa estratégia metodológica tem-lhes acarretado a crítica que, do ponto de vista positivista, tem sido apontado a estas metodologias -acusados de não serem exatamente académicos, excessivamente ideológicos e teóricos, se basearem em metodologias pobres, por fazerem um limitado uso e análise de dados empíricos e centrarem o seu foco em conceitos abstratos como ''estruturas sociais'' ou por assumirem uma «ordem do poder de género» (Pereira 2012 Lykke (2018). A validade externa não constitui, com efeito, a preocupação central na investigação qualitativa, pois o objetivo principal não é a generalização dos resultados, mas a validade interna.…”

“…Com efeito o objetivo da cientometria é gerar uma caracterização quantitativa da atividade científica e medir desse modo o progresso dos campos disciplinares, isto é, interpretar os dados quantitativos (Santos e Kobashi 2009;Hernández-Pozo e Fernández-Rius 2013;Feldman e Sandoval 2018;Aksnes, Langfeldt, e Wouters 2019). O presente trabalho consiste, assim, na apresentação de ambas as revistas, desde o seu primeiro lançamento em 1999 até ao momento presente (ex aequo 1999--2019e Faces de Eva 1999-2018. As métricas sistematizadas servirão para, numa perspectiva feminista, proceder a uma análise crítica e compreensiva da dinâmica histórica e cultural em que a área de «Estudos sobre as Mulheres» se tem vindo a desenvolver, sem esquecer que as opções investigativas dos/as autores/as e as suas oportunidades de publicação poderão ser influenciadas por fatores de natureza diversa.…”