Assessing User Competence: Conceptualization and Measurement

Abstract

Organizations today face great pressure to maximize the bene its from their investments in information technology (IT). They are challenged not just to use IT, but to use it as effectively as possible. Understanding how to assess the competence of users is critical in maximizing the effectiveness of IT use. Yet the user competence construct is largely absent from prominent technology acceptance and it models, poorly conceptualized, and inconsistently measured. We begin by presenting a conceptual model of the assessment of user competence to organize and clarify the diverse literature regarding what user competence means and the problems of assessment. As an illustrative study, we then report the findings from an experiment involving 66 participants. The experiment was conducted to compare empirically two methods (paper and pencil tests versus self-report questionnaire), across two different types of software, or domains of knowledge (word processing versus spreadsheet packages), and two different conceptualizations of competence (software knowledge versus self-efficacy). The analysis shows statistical significance in all three main effects. How user competence is measured, what is measured, what measurement context is employed:all influence the measurement outcome. Furthermore, significant interaction effects indicate that different combinations of measurement methods, conceptualization, and knowledge domains produce different results. The concept of frame of reference, and its anchoring effect on subjects' responses, explains a number of these findings. The study demonstrates the need for clarity in both defining what type of competence is being assessed and in drawing conclusions regarding competence, based upon the types of measures used. Since the results suggest that definition and measurement of the user competence construct can change the ability score being captured, the existing information system (IS) models of usage must contain the concept of an ability rating. We conclude by discussing how user competence can be incorporated into the Task-Technology Fit model, as well as additional theoretical and practical implications of our research.

Understanding knowledge management system usage antecedents: An integration of social cognitive theory and task technology fit

“…Perceived task technology fit refers to the congruence among the perceived capabilities of technology, task requirements, and the competence of users with the task and the system [54]. Goodhue and Thompson [33] have developed the "technology-to-performance chain" model, in which technology utilization depends on the fit between the technology and the tasks it supports.…”

“…Furthermore, Marcolin et al [54] implied that self-efficacy is more appropriate to evaluate the degree of the task-technology fit in context of perception or subjective measurement. They take Goodhue and Thompson's point of view a step further and argue that it is the individual's characteristics that impact the perception of TTF, and not necessarily the characteristics of technology or the level of task complexity.…”

Towards a Multi-Dimensional Model of Digital Competence in Small- and Medium-Sized Enterprises

“…The IS studies on digital competence focus either on IT competence for business managers (e.g., Bassellier et al, 2001;Bassellier & Benbasat, 2004) or on a general IT user competence (e.g., Marcolin et al, 2000;Jasperson et al, 2005). In the former approach, Bassellier et al (2001) suggest that in order to become IT competent, business managers need to develop a knowledge-and experience-based tridimensional IT competence: competence as a skill, competence as a personality trait, and competence as knowledge.…”

Digital Competence: A Multi-dimensional Conceptualization and a Typology in an SME Context

“…In particular, the development of internal ICT skills combined with management's knowledge and attitudes towards ICT adoption and use create the competences required to achieve higher levels of success with ICT use in SMEs [12]. In the last two decades, much of the research in IS had adopted a more technical perspective [26] and has focused on identifying: 1) business managers technical skills [5]; 2) ICT specialists knowledge and skill [32]; or 3) ICT professionals' personality characteristics [3]. One thing that should be highlighted, is that most past IS studies on individual DC had a relatively narrow and specific conceptualization of user [25].…”

“…Some conceptualizations tend to emphasize the practical and technical aspects of using ICT [26], while others suggest that developing DC necessitates a focus on the acquisition of higher order thinking skills [15] in various areas [9]. ICT are more and more ubiquitous and the use of such technology is now spread across the different types of jobs, organizations' levels and to accomplish a multitude of various tasks.…”