Evaluating a multimedia authoring tool

John, Bonnie E.; Mashyna, Matthew M.

doi:10.1002/(sici)1097-4571(199711)48:11<1004::aid-asi4>3.0.co;2-#

Cited by 18 publications

(9 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One is to match descriptions of usability problems at varying degrees of similarity. John and Mashyna (1997) distinguished a precise and a vague hit between usability problems; Connell et al (2004) similarly distinguished a hit from a possible hit. While matching in degrees reduces part of the uncertainty of matching, it is not a general solution because it does not remove most of the difficulties discussed above.…”

Section: Related Worksupporting

confidence: 61%

Comparison of techniques for matching of usability problem descriptions

Hornbæk

Frøkjær

2008

Interacting with Computers

View full text Add to dashboard Cite

a b s t r a c tMatching of usability problem descriptions consists of determining which problem descriptions are similar and which are not. In most comparisons of evaluation methods matching helps determine the overlap among methods and among evaluators. However, matching has received scant attention in usability research and may be fundamentally unreliable. We compare how 52 novice evaluators match the same set of problem descriptions from three think aloud studies. For matching the problem descriptions the evaluators use either (a) the similarity of solutions to the problems, (b) a prioritization effort for the owner of the application tested, (c) a model proposed by Lavery and colleagues [Lavery, D., Cockton, G., Atkinson, M.P., 1997. Comparison of evaluation methods using structured usability problem reports. Behaviour and Information Technology, 16 (4/5), 246-266], or (d) the User Action Framework [Andre, T.S., Hartson, H.R., Belz, S.M., McCreary, F.A., 2001. The user action framework: a reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54 (1), 107-136].The resulting matches are different, both with respect to the number of problems grouped or identified as unique, and with respect to the content of the problem descriptions that were matched. Evaluators report different concerns and foci of attention when using the techniques. We illustrate how these differences among techniques might adversely influence the reliability of findings in usability research, and discuss some remedies.

show abstract

Section: Related Worksupporting

confidence: 61%

Comparison of techniques for matching of usability problem descriptions

Hornbæk

Frøkjær

2008

Interacting with Computers

View full text Add to dashboard Cite

show abstract

“…John and Mashyna (1997) provided examples taken from studies comparing evaluation methods at the problem type level, without data on problem tokens or where problem types cover disparate problem tokens. They argued that these studies find a large overlap between sets of problems, possibly leading to an overrating of the success of methods.…”

Section: Dogma No 2: Matching Problem Descriptions Is Straightforwardmentioning

confidence: 99%

“…Studies have also suggested that the extent to which evaluators are guided by heuristics is uncertain (Doubleday et al 1997): 'It is difficult to know how far the heuristics guided the evaluators, even though they were given a detailed synopsis of the meaning of each heuristic' (p. 107). John and Mashyna (1997) found, similarly to the study by Jeffries quoted above, that only about two-thirds of the problems found during a cognitive walkthrough were actually that the evaluator attributes to the technique. Regarding empirical evaluation methods, several studies suggest that think-aloud practice differs markedly from prescriptions (e.g.…”

Section: '[T]he User Action Framework Allows Usability Engineers To Nmentioning

confidence: 99%

Dogmas in the assessment of usability evaluation methods

Hornbæk

2010

Behaviour & Information Technology

View full text Add to dashboard Cite

Usability evaluation methods (UEMs) are widely recognised as an essential part of systems development. Assessments of the performance of UEMs, however, have been criticised for low validity and limited reliability. The present study extends this critique by describing seven dogmas in recent work on UEMs. The dogmas include using inadequate procedures and measures for assessment, focusing on win-lose outcomes, holding simplistic models of how usability evaluators work, concentrating on evaluation rather than on design and working from the assumption that usability problems are real. We discuss research approaches that may help move beyond the dogmas. In particular, we emphasise detailed studies of evaluation processes, assessments of the impact of UEMs on design carried out in real-world systems development and analyses of how UEMs may be combined.

show abstract

“…To identify UPs in three aspects: game interface (e.g., button, navigation), game mechanism (e.g., ease of use/control, learnability), and gameplay (e.g., goal, responsiveness). Criteria for UPs were applied (John and Mashyna, 1997).…”

Section: Section 3: Adaptivitymentioning

confidence: 99%

“…Usability: Observations were a crucial technique for identifying usability problems (UPs), and traditional criteria for UPs were applied (John and Mashyna, 1997). A preliminary list of UPs was verified against the videorecordings.…”

Section: Revisit Of Four-dimensional Evaluation Frameworkmentioning

confidence: 99%