Commercial competence: comparing test results of paper-and-pencil versus computer-based assessments

Abstract

Background: Vocational education and training (VET) aims to enable young adults or trainees to participate in the workplace, and to promote their vocational capacities. In order to examine trainees' competencies at the end of VET, appropriate instruments are needed. This contribution aims: (1) to give an outline of such an instrument, one that has been designed to evaluate vocational competencies in the field of economics, and (2) to present the results of an empirical comparison of two possible test modes: computer-based assessment (CBA) versus paper-based assessment (PBA). The use of new technologies offers various opportunities for competence measurement: in particular, the computer as an assessment tool presents an authentic work tool drawn from professional life and promises novel ways of designing assessments. However, the current assessment practice in Germany is dominated by the use of traditional PBA, and there is less evidence about the possible effects of CBA. This study addresses the question of whether there are significant differences in the various ways of representing and measuring commercial competence with respect to specific content, item format, and, finally, motivational aspects. Methods:A sample of 387 trainees from the German VET system was used to compare these two kinds of assessment. The analyses were realized using Item Response Theory and, particularly, Differential Item Functioning to detect differences between PBA and CBA at the item level. In addition to the performance data, motivational aspects, such as emotional state and test attractiveness, were also taken into account by a pre-/post-questionnaire. Results:The study demonstrates that both test formats (CBA and PBA) can represent commercial competence in a valid and reliable way, but differences were found for certain items in the number of correct responses. The PBA shows a slight advantage in respect of overall item and model fit. Another key finding of our comparative study, at item level, is important from an instructive viewpoint: (domain) specific items are easier to solve in CBA than in PBA, whereas more general items are answered correctly more frequently in the latter. Contrary to expectations, we could not confirm the overall dominance of CBA against PBA on the basis of test takers' motivation, but values from CBA were more stable over time. Conclusions:The study facilitated making the strengths and weaknesses of both test formats evident, and this implies the possibility of identifying opportunities for further development in assessment practice and in designing tests. Selected design criteria and aspects of test administration are discussed, with the aim of seeking to optimize Open Access © The Author(s) 2017. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the s...

Challenges in using authentic assessment in 21st century ESL classrooms

“…As such, school administration should play a facilitating role in giving the teachers a helping hand, in this case, the excellent teachers, to lessen their burden caused by overwhelming documentation. Interestingly, scholars also suggested that digitisation of the assessment of the task could be carried out in the 21st-century as it will allow for the learners to be tested more vigorously using the state-of-the-art methods [40,41].…”

Automotive Electrical Learning Module to Improve Students Interest and Learning Achievement of Vocational School

“…[15]. This technique is used to find out whether certain scores are significantly different or not with the median of a sample [14]. The output of tone sample test on learning interest shows that the class used as experiment scored a mean of 57.00, which means there is a difference in learning interests by using a better module categorized as Advanced.…”

“…Based on the t test counting result of one sample test SPSS program that aims to differentiate [14], this technique is used to test the output counting of test of normality [10], which shows the significance score result of control class for 0.200 and significance score of experimental class for 0.118. Therefore, the second result is considered normal.…”

Computerized testing in reading comprehension skill: investigating score interchangeability, item review, age and gender stereotypes, ICT literacy and computer attitudes

“…Furthermore, in universities, remote learning and assessment through free software web conferencing systems such as BigBlueButton is being used, and the exams are usually conducted through computer, because some features of software may not be shown in mobile. Score interchangeability between CFLT and PPBT as the controversial issue in the last decade (Sangmeister, 2017) in Computer-Based Testing field should be investigated empirically through studying psychometric properties (Burr et al, 2016) as the prerequisite of replacing CFLT with PPBT (Khoshsima et al, 2019;Rausch, Seifried, Wuttke, Kogler and Brandt, 2016). Therefore, since authenticating and sustaining validity and reliability of measurement is essential for replacing CFLT with PPBT, the current study concentrated on whether validity and reliability measurement would be violated by changing testing administration mode and whether scores received from PPBT and CFLT would be interchangeable or equivalent (TEA, 2008).…”