A comparison of techniques for measuring sensemaking and learning within participant‐generated summaries

Abstract

While it is easy to identify whether someone has found a piece of information during a search task, it is much harder to measure how much someone has learned during the search process. Searchers who are learning often exhibit exploratory behaviors, and so current research is often focused on improving support for exploratory search. Consequently, we need effective measures of learning to demonstrate better support for exploratory search. Some approaches, such as quizzes, measure recall when learning from a fixed source of information. This research, however, focuses on techniques for measuring open-ended learning, which often involve analyzing handwritten summaries produced by participants after a task. There are two common techniques for analyzing such summaries: (a) counting facts and statements and (b) judging topic coverage. Both of these techniques, however, can be easily confounded by simple variables such as summary length. This article presents a new technique that measures depth of learning within written summaries based on Bloom's taxonomy (B.S. Bloom & M.D. Engelhart, 1956). This technique was generated using grounded theory and is designed to be less susceptible to such confounding variables. Together, these three categories of measure were compared by applying them to a large collection of written summaries produced in a task-based study, and our results provide insights into each of their strengths and weaknesses. Both fact-to-statement ratio and our own measure of depth of learning were effective while being less affected by confounding variables. Recommendations and clear areas of future work are provided to help continued research into supporting sensemaking and learning.

Investigation of users' knowledge change process in learning‐related search tasks

“…Research in search as learning has proposed several ways to measure the learning outcome after searching (Bhattacharya & Gwizdka, ; Collins‐Thompson, Rieh, Haynes, & Syed, ; Gadiraju, Yu, Dietze, & Holtz, ; Wilson & Wilson, ; Yu et al, ); and the relationship between search interactions and learning performance or knowledge change (Bhattacharya & Gwizdka, ; Ghosh, Rath, & Shah, ; Liu & Song, ).…”

Assessing Learning Outcomes in Web Search

“…Summary quality was evaluated based on topic-specific criteria including the number of reasonable topics, overall quality of the topic description, and number of arguments. Wilson and Wilson [25] developed systematic techniques to measure the depth of learning at three levels: quality of facts, interpretation of data into statements, and use of critique. These previous studies indicate that traditional measures in information retrieval, such as recall and precision, can be effectively complemented with alternative measures that pay more attention to search behavior, process, and task outcomes beyond basic search results.…”

The Tetris Model of Resolving Information Needs within the Information Seeking Process

“…The interesting research questions lie around the impact that incomplete prior knowledge has on assessing whether new information correctly resolves an information need. Answer correctness, or a measure of an answers strength, is often used in studies of searching behaviour, whilst also trying to measure pre-and post-search knowledge on a topic [55]. The Tetris Model may help to draw new finer-grained questions about the sequence in which information is found during search.…”