Usability evaluation considered harmful (some of the time)

Abstract

Current practice in Human Computer Interaction as encouraged by educational institutes, academic review processes, and institutions with usability groups advocate usability evaluation as a critical part of every design process. This is for good reason: usability evaluation has a significant role to play when conditions warrant it. Yet evaluation can be ineffective and even harmful if naively done 'by rule' rather than 'by thought'. If done during early stage design, it can mute creative ideas that do not conform to current interface norms. If done to test radical innovations, the many interface issues that would likely arise from an immature technology can quash what could have been an inspired vision. If done to validate an academic prototype, it may incorrectly suggest a design's scientific worthiness rather than offer a meaningful critique of how it would be adopted and used in everyday practice. If done without regard to how cultures adopt technology over time, then today's reluctant reactions by users will forestall tomorrow's eager acceptance. The choice of evaluation methodology -if any -must arise from and be appropriate for the actual problem or research question under consideration.

User studies in cartography: opportunities for empirical research on interactive maps and visualizations

“…How should we assess the quality and impact of transferable and contextual insights generated from UCD studies? Can we conduct design studies to simultaneously approach practical and theoretical goals, and at what point does doing so become harmful to the creative development process (Greenberg & Buxton, 2008)?…”

“…But, which empirical methods should we use, and at what times during design? Greenberg and Buxton (2008) suggest that the wrong methods applied at the wrong time in a tool's development actually can be harmful to the design of the tool. In particular, they note that focusing on usability issues too early in the development of a tool might lead to 'getting the design right' at the expense of 'getting the right design' (Greenberg & Buxton, 2008, p. 115).…”

Perspectives for Defining Student Teacher Performance-Based Teaching Skills Indicators to Provide Formative Feedback through Learning Analytics

“…High quality input data is considered as a prerequisite for the successful functioning of LA in many analytical articles (e.g. Dringus, 2012;Greenberg & Buxton, 2008;Greller & Drachsler, 2012). As Clow (2013) explains:…”

HCI and innovation

“…How can long term relationships between users, researchers and designers be set up to improve design [23]? At what level of granularity should usability evaluations be done so as not to stifle promising design ideas and forestall creative adoption within a culture [16]? This last point led Greenberg and Buxton to consider the poor usability of successful innovations such as the radio and early automobiles, which was forgiven by the market because of their great usefulness and long term potential.…”