Effective and efficient measurement of the development of skill and knowledge, especially in domains of human activity that involve complex and challenging problems, is important with regard to workplace and academic performance. However, there has been little progress in the area of practical measurement and assessment, due in part to the lack of automated tools that are appropriate for assessing the acquisition and development of complex cognitive skills. In the last 2 years, an international team of researchers has developed and validated an integrated set of assessment tools called highly integrated model assessment technology and tools (HIMATT) which addresses this deficiency. HIMATT is web-based and has been shown to scale up for practical use in educational and workplace settings, unlike many of the research tools developed solely to study basic issues in human learning and performance. In this paper, we describe the functions of HIMATT and demonstrate several applications for its use. Additionally, we present two studies on the quality and usability of HIMATT. We conclude with research suggestions for the use of HIMATT and for its further development.

No abstract

Our study integrates automated natural language-oriented assessment and analysis methodologies into feasible reading comprehension tasks. With the newly developed T-MITOCAR toolset, prose text can be automatically converted into an association net which has similarities to a concept map. The ''text to graph'' feature of the software is based on several parsing heuristics and can be used both to assess the learner's understanding by generating graphical information from his or her text and to generate conceptual graphs from text which can be used as learning materials. In this study we investigate the effects of association nets made available to learners prior to reading. The results reveal that the automatically created graphs are highly similar to classical expert graphs. However, neither the association nets nor the expert graphs had a significant effect on learning, although the latter have been reported to have an effect in previous studies.

No abstract

Research on learning via system‐dynamics based learning environments depends on accurate measurement of learning. Most such research considers at least two aspects of learning: participants' understanding of the models and problems; and participants' performance in the environment, i.e. quality of decision making. Understanding is much more difficult to measure than performance. Measurement of understanding is often done by eliciting verbal protocols from participants. Coding and analysis of verbal protocols are subjective and time consuming. To facilitate measurement and analysis of understanding via verbal protocols, we investigate the utility of a software application which performs such analysis automatically. We assess this automated analysis methodology using data from a system‐dynamics based learning environment. The results from the automated analysis are comparable to the results from a manual analysis when we analyze how participants' understanding compares to experts and how it changes over time. Additionally, the automated analysis facilitates investigating relationships between understanding and performance. Copyright © 2012 System Dynamics Society.