User-centered evaluation of adaptive and adaptable systems: a literature review

Abstract

This literature review focuses on user-centered evaluation (UCE) studies of adaptive and adaptable systems. Usability, perceived usefulness and appropriateness of adaptation are the three most commonly assessed variables. Questionnaires appeared to be the most popular method, followed by interviews and data log analysis. The quality of most questionnaires was questionable, and the reporting of interviews and think-aloud protocols was found to be shallow. Furthermore, data logs need triangulation in order to be useful. The reports encountered lack empirical value. The article models the iterative design process for adaptive and adaptable systems, linked to the goals of UCE: supporting decisions, detecting problems and verifying quality. This model summarizes the variables to be assessed at each stage of the process and the relevant methods to assess them.

State of The Art of Learning Styles-Based Adaptive Educational Hypermedia Systems (Ls-Baehss)

“…Adaptive system can be defined as implicitly inferring users' preferences by observing their interaction with a system in order to tailor its output. Adaptable systems, on the other hand, means providing explicit input in order to personalize output [7]. Brusilovsky [2] identified the main criteria of adaptive hypermedia systems (AHSs) as follows: "it should be a hypertext or hypermedia system; it should have a user model; it should be able to adapt the hypermedia using this model".…”

Factors affecting health care workers’ adoption of a website with infection control guidelines

“…Target group [26] + The website is useful for nurses (n = 12), [27] + The website is useful for patients (n = 14) [28] + The website is useful for patients' relatives (n = 6) [29] + The website is useful for physicians (n = 6) Enabling (work-related) (53 statements) Workload [30] + The website lowers work pressure for infection control professionals, as they will receive fewer questions from health care workers (n = 3) [31] − High work pressure impedes consulting the website during working hours (n = 9) Tacit knowledge [32] + The website enables health care workers with lack of knowledge of MRSA guidelines to retrieve relevant information (n = 4) [33] − Personal, practical, and experiential wisdom about MRSA guidelines leads health care workers to not need the website (n = 10)…”

“…Behavioral intention (20 statements) [54] + I will (continue) to use the website (n = 12) [55] − I will not (continue to) use the website (n = 4), [56] − I do not know yet whether I will (continue to) use the website (n = 4) can only confirm known variables and assess their scores [29]. Since we were primarily interested in exploring the practical meaning HCWs gave to theoretical concepts in order to tailor adoption-enhancing intervention strategies to HCWs' current realm, rather than testing a number of propositions, we opted to conduct interviews with open-ended questions.…”

Evaluating the Adaptation of a Learning System before the Prototype Is Ready: A Paper-Based Lab Study

“…This is also reflected in the substantial amount of evaluations that have been published so far. Van Velsen et al [1] present an overview and have noted several limitations in current evaluation practices. A variety of evaluation frameworks have been presented [2], [3], [4], all of which propose to break down the adaptive system into assessable, selfcontained functional units.…”

“…It is recommended that these two research questions are investigated in an experimental setting using a running system (or prototype) where the algorithms are already implemented [1], [3]. The problem is that in many situations the development cycle of the software product is short and the evaluation might become obsolete as soon as a new version has been developed [4].…”