An Investigation of Metrics for the In Situ Detection of Software Expertise

Abstract

Task-based analysis is a common and effective way to measure expertise levels of software users. However, such assessments typically require inperson laboratory studies and inherently require knowledge of the user's task. Today, there is no accepted method for assessing a user's expertise levels outside of a lab, during a user's own home or work environment activities. In this article, we explore the feasibility of software applications automatically inferring a user's expertise levels, based on the user's in situ usage patterns. We outline the potential usage metrics that may be indicative of expertise levels and then perform a study, where we capture such metrics, by installing logging software in the participants' own workplace environments. We then invite those participants into a laboratory study and perform a more traditional task-based assessment of expertise. Our analysis of the study examines if metrics captured in situ, without any task knowledge, can be indicative of user expertise levels. The results show the existence of significant correlations between metrics calculated from in situ usage logs, and task-based user expertise assessments from our laboratory study. We discuss the implications of the results and how future software applications may be able to measure and leverage knowledge of the expertise of its users.

Visual design and online shopping experiences: When expertise allows consumers to refocus on website attractiveness

“…As with Study 1, Study 2 involved questions about browsing one of two web interfaces depending on the experimental condition. Our measure of user expertise reflects the definitions provided by Alba and Hutchinson (1987) and Grossman and Fitzmaurice (2015) and is consistent with prior studies (e.g. Demangeot and Broderick, 2010;Roehm and Sternthal, 2001;Sujan, 1985).…”

“…Human–computer interaction (e.g. Grossman and Fitzmaurice, 2015; Lawson et al, 2009) and information system (e.g. Aljukhadar and Senecal, 2016; Sedera and Dey, 2013) literature widely discusses the user expertise concept, with such expertise defined as ‘a user’s ability to complete tasks with the system’ (Grossman and Fitzmaurice, 2015: 472).…”

Tutorial mismatches: understanding the frictions due to interface differences when following software video tutorials

“…Design recommendation 6: Spatial proximity should be favored for inexperienced users (focusing on improving the learnability of an interface), while direct reachability should be favored for experienced ones (focusing on performance improvement). In addition, this would require to have models capable of reliably assess user expertise [29] (in this case, mostly UI, command, and task expertise) in order to tailor instructional videos to user's level.…”

Design visuel et expérience d’achat en ligne : quand l’expertise permet aux consommateurs de se (re)centrer sur l’attractivité du site d’e-commerce

“…L’expertise dans l’utilisation des produits numériques est plus difficile à acquérir lorsqu’elle implique des coûts d’apprentissage élevés (Lakshmanan et Krishnan, 2011 ; Monnot, 2020). L’interaction homme-machine (e.g., Grossman et Fitzmaurice, 2015 ; Lawson et al, 2009) et les systèmes d’information (e.g., Aljukhadar et Senecal, 2016 ; Sedera et Dey, 2013) discutent largement du concept d’expertise de l’utilisateur, qui se définit comme « la capacité d’un utilisateur à accomplir des tâches avec le système » (Grossman et Fitzmaurice, 2015 : 472).…”

“…Comme pour l’étude 1, l’étude 2 impliquait des questions sur la navigation dans l’une des deux interfaces Web en fonction de la condition expérimentale. La mesure de l’expertise des utilisateurs reflète les définitions fournies par Alba et Hutchinson (1987) et Grossman et Fitzmaurice (2015), et est conforme aux études antérieures (e.g., Demangeot et Broderick, 2010 ; Roehm et Sternthal, 2001 ; Sujan, 1985). Un score d’expertise utilisateur a été calculé en totalisant le score de chaque participant sur chacune des 12 questions posées (quatre questions pour chacune des trois différentes tâches de navigation), avec une réponse correcte codée 1 et une réponse incorrecte codée 0.…”