Against method: Exploding the boundary between qualitative and quantitative studies of science

Abstract

Quantitative and qualitative studies of science have historically played radically different roles with opposing epistemological commitments. Using large-scale text analysis, we see that qualitative studies generate and value new theory, especially regarding the complex social and political contexts of scientific action, while quantitative approaches confirm existing theory and evaluate the performance of scientific institutions. Large-scale digital data and emerging computational methods could allow us to refigure these positions, turning qualitative artifacts into quantitative patterns into qualitative insights across many scales, heralding a new era of theory development, engagement, and relevance for scientists, policy-makers, and society.

Semantic and relational spaces in science of science: deep learning models for article vectorisation

“…Kozlowski et al (2019) suggest to use analogies for the description of social dimensions. Kang and Evans (2020) applied this to the field of Science of Science. The authors defined two word embeddings and compare how different concepts, like 'theory' and 'measure' were projected into the 'good-bad' dimension, based on a selection of journals assigned to the quantitative and qualitative research communities.…”

“…An embedding is a low dimensional dense vector, i.e., of real-valued representations that encodes relevant information from the original, high dimensional data (Mikolov et al 2013a). Nevertheless, not much research has been carried out to bring these techniques closer to the field of Science of Science, except of the following examples: A recent article used Word2Vec (Mikolov et al 2013a) to distinguish the most relevant terms in quantitative and qualitative research in the field of Science of Science (Kang and Evans 2020). Paper2Vector (Zhang et al 2019) is a model that trains word-word, document-word, and document-document relations based on the Skip-Gram model (Mikolov et al 2013a).…”

“…The article focuses on document-level embeddings, based on the semantic and relational aspects of articles using NLP and GNN's to explore semantic and relational spaces in Science of Science. Four different aspects will be analysed: (a) collaboration-patterns (Sooryamoorthy 2009; b) the cumulative effect on citations, i.e., the Matthew effect in science (de Solla Price 1963;Garfield 1972;Garfield and Merton 1979); (c) the position of countries in science global production (King 2004); and (d) the quantitative-qualitative divide (Kang and Evans 2020).…”

NERO: A Biomedical Named-entity (Recognition) Ontology with a Large, Annotated Corpus Reveals Meaningful Associations Through Text Embedding

“…Building on these capacities 10,14,15 , we proposed a general method for constructing meaningful dimensions by taking the arithmetic mean of word vectors representing antonyms along a dimension and using them to diagnose their meanings. This approach has been widely validated 15–19 , and we employed it here to construct and compare the meanings embedded in NERO and our annotated corpus with ground truth data about drugs and diseases. In order to evaluate word embeddings based on NERO, we identified two disease properties —(1) severity and (2) gender specificity—and likewise two therapeutic drug properties —(1) toxicity and (2) expense—not directly present in text, but highly relevant to diagnosis and treatment, and on which text-independent ground truth data exists.…”

“Garbage in, garbage out” revisited: What do machine learning application papers report about human-labeled training data?

“…Our project is based on the methodology of structured content analysis, which seeks to systematically turn qualitative phenomenon into categorical and quantitative data (Riff et al, 2013). We draw on and situate our study within the growing efforts to bridge the fields of qualitative and quantitative science studies (Leydesdorff et al, 2020;Bowker, 2020;Cambrosio et al, 2020;Kang and Evans, 2020). Quantitative science studies often examines the outputs of science, such as analyzing bibliometrics and other already-quantitative trace data to understand how scientists' final products have been received within science and other institutions.…”