Bridging the Evaluation Gap

Wouters, Paul

doi:10.17351/ests2017.115

Cited by 18 publications

(13 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…My point is that these stories, while reassuring in some ways, belie containment as "survival narratives" given the complexity of strategies for and observations about such processes. Together these two contributions (Bal 2017;Wouters 2017) are likely to generate reflections about patterns of compliance and resistance and about the relations within collectives that may shift orientations in either direction (indeed, how we discern these directions). These accounts also raise questions, with due attention to situational specificity (which virtually all these authors advocate) about where, when, and how to resist and contest indicator regimes.…”

Section: Survival Narratives: Sts Case Studiesmentioning

confidence: 96%

“…But the authors are also often quite open about their more positive emotions. Ulrike Felt (2017), for example, explores "the moments of empowerment and pleasure" enjoyed by her and others resulting from apparent successes in playing the appraisal game (see also Bal 2017;Irwin 2017;Wouters 2017). There are even references to some finding "joy in the indicator game" (Fochler and de Rijcke 2017, 23, 30).…”

Section: Being Implicated / Implicated Beingsmentioning

confidence: 99%

“…Moreover, there are anecdotes within some of the essays (Felt 2017;Irwin 2017) and two articles in particular--by Roland Bal and Paul Wouters (Bal 2017;Wouters 2017) which constitute what might be called "survival narratives": participants' stories of institutional units which (at least to the point registered by the authors) have weathered the slings and arrows of sometimes outrageous appraisal. These two contributions comprise, as Roland notes, "auto-ethnographic" case studies and, in their rather small-scale versions, they could be linked to the lauded STS tradition of ethnographic laboratory studies.…”

Section: Survival Narratives: Sts Case Studiesmentioning

confidence: 99%

See 2 more Smart Citations

Implicated in the Indicator Game: A Response to the Debate

McNeil

2017

Engaging STS

View full text Add to dashboard Cite

Let me begin by saying how much I welcome this collection. It is to be celebrated, as Alan Irwin (2017) encourages us to do, as exemplifying the strengths of STS as a field and the resourcefulness of its community. It is refreshing to find scholars reflecting on the structures and operations of the academic world and on their own practices, trying to develop strategies in the face of the growing pressures and consequences of the "indicator game." What makes this set of pieces distinctive in a growing literature on these developments (much of which is deployed and referenced in this issue) is that the authors try to think through what STS as a field and as a set of practices with a diverse theoretical repertoire might have to offer in the struggles to live with and against these developments. Their STS thinking is articulated with direct reference to specificities of the authors' own experiences. They offer situated reflections about doing STS and using it in dealing with "the culture of competition and quantification" (Müller 2017) which increasingly pervades academic life.My job as I understand it is to respond to these initiatives. I have chosen to do this, in the positive spirit advocated by many of the contributors (especially, but not only, Irwin 2017), highlighting what I consider to be key features of the collection. In so doing I will also make my own suggestions about further challenges for STSers (a borrowed neologism) and possibly others as we navigate this difficult terrain. Being Implicated / Implicated BeingsThe editors' epitaph (Fochler and de Rijcke 2017, 22) for the collection is striking: a quotation from Marilyn Strathern cautioning that, in dealing with the indicator game, "we are witnessing an effect that we (practitioners in higher education) have helped produce. Auditors are not aliens: they are a version of ourselves" (Strathern 1997, 319.) Strathern's "auditors-R-us" caution haunts the project and a number of the authors mull it over. The collection is up-front in acknowledging that STSers and other academics not only get caught up in the indicator mechanisms, but turn (even sometimes are) their cogs--enabling them to run.Some contributors flesh out how enrolments in these systems are solicited and ensured emotionally and psychically. There are allusions to the anxieties which prompt compliance, in

show abstract

Section: Survival Narratives: Sts Case Studiesmentioning

confidence: 96%

Section: Being Implicated / Implicated Beingsmentioning

confidence: 99%

Section: Survival Narratives: Sts Case Studiesmentioning

confidence: 99%

See 1 more Smart Citation

Implicated in the Indicator Game: A Response to the Debate

McNeil

2017

Engaging STS

View full text Add to dashboard Cite

show abstract

“…Paul Wouters' (2017) essay is concerned with how to bridge the "evaluation gap" that many scholars experience as the criteria used in assessment do not seem to adequately represent what they themselves value in their work. He discusses the recent evaluation of his own center as an attempt to bridge this very gap, and reminds us that it is productive to see evaluations not as the (obviously impossible) attempt to produce a true representation of past work, but as explorations of "who one wants to be."…”

Section: Reflecting the Debatementioning

confidence: 99%

“…The goal is to simultaneously play the game, expand the board, and diversify the pool of potential players. It is visible in attempts to turn institutional assessments into experimental, intellectually motivated future-making exercises instead of conservative adjudications of past performance (Wouters 2017). It is also visible in attempts to enhance the brand of open access, non-profit journals in our field through JIFqualification (Vann 2017).…”

Section: Reflecting the Debatementioning

confidence: 99%

<b>Implicated in the Indicator Game? An Experimental Debate</b>

Fochler

Rijcke

2017

Engaging STS

View full text Add to dashboard Cite

The rise of new modes evaluating academic work has substantially changed institutions and cultures of knowledge production. This has been reflected and criticized in the literature in STS and beyond. For STS scholars, these debates (should) however have an even more specific dimension. Many of us are experts on aspects of these changes. But at the same time, we too are part of the processes we are analyzing, and often criticizing. To put it slightly provocatively, often we cannot avoid playing the very same game that we scrutinize. This creates tensions that many of us reflect on, and it certainly has created many implicit and explicit normative stances on how to deal with them. Yet it seems that so far there has been little room in our field to reflect on and exchange this particular kind of experience-based knowledge. There are many different ways to engage with the dynamics of evaluation, measurement and competition in contemporary academia, or to play what we refer to colloquially here as the "indicator game." With this debate, we would like to give room to the expression and discussion of some of these ways. This text is the introduction and prompt to an experimental debate. We discuss the state of the academic discussion on the impact of indicator-based evaluation on academic organization, epistemic work and identities. We use insights from these debates to raise questions for how STS and STSers themselves deal with the indicator game. In conclusion, we summarize our contributors' arguments and propose the concept of "evaluative inquiry" as a new way of representing the quality of STS work in evaluative contexts.

show abstract

Your Brother’s Gatekeeper: How Effects of Evaluation Machineries in Research Are Sometimes Enhanced

Dahler‐Larsen

2022

Peer Review in an Era of Evaluation

View full text Add to dashboard Cite

Many warnings are issued against the influence of evaluation machineries (such as bibliometric indicators) upon research practices. It is often argued that human judgment can function as a bulwark against constitutive effects of evaluation machineries. Using vignettes (small case narratives) related to the Danish Bibliometric Research Indicator (BRI), this chapter shows that gatekeepers who “know the future” and use this “knowledge” in a preemptive or precautionary way play a key role in the construction of reality which comes out of the BRI. By showing that human judgment sometimes enhances or multiplies the effects of evaluation machineries, this chapter contributes to an understanding of mechanisms which lead to constitutive effects of evaluation systems in research.

show abstract

Bridging the Evaluation Gap

Cited by 18 publications

References 14 publications

Implicated in the Indicator Game: A Response to the Debate

Implicated in the Indicator Game: A Response to the Debate

<b>Implicated in the Indicator Game? An Experimental Debate</b>

Your Brother’s Gatekeeper: How Effects of Evaluation Machineries in Research Are Sometimes Enhanced

Contact Info

Product

Resources

About