Conceptualizing the role of evaluation systems in markets: The case of dominant evaluators

Abstract

Abstract. Evaluation is usually an internalized process that is intrinsic to the activities

Give me a two-by-two matrix and I will create the market: Rankings, graphic visualisations and sociomateriality

“…The starting point for this paper was the suggestion that these measures wield forms of influence that have yet to be identified by existing forms of analysis. Whilst there are a growing number of studies that analyse the power of rankings, some from within Accounting research (Free et al 2009, Kornberger and Carter 2010, Jeacle and Carter 2011, others from outside this area (Shrum 1996, Wedlin 2006, Blank 2007, Espeland and Sauder 2007, Author Study 2009, Karpik 2010, Kwon and Easton 2010, very few have provided insights into their makeup and minutiae (but see Schultz et al [2001] who point to some aspects of their construction). One implication when a crucial market mechanism is black-boxed is that we only ever develop a partial understanding of its constitutive capacity.…”

“…The 'top 10 MBA programmes' can (and often are) extended to include the 'top 50', 'top 100' degrees, for instance. Kwon and Easton (2010), in their discussion of the Financial Times' list of MBA programmes, suggest that the longer the list the more comprehensive or 'global' it may appear in certain peoples' eyes: "…individual consumers can find comfort in the perception that they can choose the 'best' among hundreds or thousands of alternatives, rather than the 'best' among several 'good enough' alternatives arising through the search process. The FT MBA 100 allows buyers to maximize their choice of a highly ranked school, given personal constraints such as budget, geographical preferences and entry requirements" (ibid., 133).…”

“…Whilst rankers have not been particularly innovative with regard to methodology, or how assessments are put together, they have been at the forefront in terms of developments in 'format and presentation'. Kwon and Easton (2010) describe how the Michelin Red Guide, for instance, was amongst the first of the major rankers to supplement complicated forms of quantitative data with 'qualitative descriptors'. It rated restaurant quality by producing the "now famous three-star scale to denote relative excellence" (ibid.…”

“…The fact our project was funded filled us with both excitement and (it must be said) a certain amount of dread! There is a perception that it is difficult to gain access to Gartner (a point said to be true of rankers more generally [Kwon and Easton 2010], which perhaps explains the paucity of studies on the production of rankings). Nevertheless, we set out to conduct fieldwork in the hope that we would get lucky (and 'fortune' does seem to feature in a lot of research).…”

IT Industry Analysts: A Review and Two Research Agendas

“…They also provide a means to commoditize complex IT products and services through the commensuration and the reduction of their features to comparable and similar metrics [Pollock and Williams, 2009]. Yet, what is particular about the IT innovations marketplace is that it is one of the few business-to-business marketplaces where third-party actors play a significant role in the mediation of exchanges by delineating product categories, by certifying vendors' characters and products, and by acting as advisor to buyers [Kwon and Easton, 2010]. Such third-party actors are commonplace in many consumer products and cultural markets, but, in contrast to IT industry analysts, they do not tend to exhibit as much clout concentrated in the hands of a few actors.…”