Background: To measure higher-order outcomes of vocational education and training (VET) we developed a computer-based assessment of domain-specific problemsolving competence. In modeling problem-solving competence, we distinguish four components of competence: (1) knowledge application, (2) metacognition, (3) self-concept, and (4) interest as well as thirteen facets of competence, each of which is assigned to one of the four components.
Methods:With regard to ecological and content validity, rather than apply highly structured items (e.g. multiple choice items), we developed three authentic problem scenarios and provided an open-ended problem space in terms of an authentic office simulation. The assessment was aimed at apprentice industrial clerks at the end of a 3-year apprenticeship program and focused on the domain of controlling (i.e., support of managerial decisions, cost planning, cost control, cost accounting, etc.). The computer-based office simulation provided typical tools (e.g., email client, spreadsheet software, file system, notebook, calculator, etc.). In order to assess the non-cognitive components in our competence model, we implemented an integrated measurement of self-concept and interest that we refer to as 'Embedded Experience Sampling' (EES). Test-takers are requested to spontaneously answer short prompts (EES items) during the test that are embedded in typical social interactions in the workplace. The empirical section is based on a study with 780 VET students from three commercial training occupations in Germany (industrial clerks and apprentices from two similar VET programs). The focus of the contribution is on testing a theoretically derived competence model based on item response theory, the implemented scoring methods and reliability of the instrument. Fine-grained response patterns from automated codings and human ratings were condensed into one partial credit item for each scenario and each of the facets in the cognitive component 'knowledge application' .
Results:The multidimensional Rasch analysis revealed satisfactory EAP/PV reliabilities, which are between .78 and .84 for the 'knowledge application' facets and between .77 and .85 for the non-cognitive facets. Furthermore, the achievement differences between the industrial clerks and their comparison groups are as assumed.
Conclusions:In our study, we introduced an innovative method to measure noncognitive facets of problem-solving competence in the course of complex problem Rausch et al. Empirical Res Voc Ed Train (2016) Rausch et al. Empirical Res Voc Ed Train (2016) 8:9 scenarios. Furthermore, by using authentic problem scenarios and providing an openended and authentic problem space, our assessment of domain-specific problem-solving comeptence focuses on ecological validity but also ensures reliability.
RESEARCH