A Multi-Method Information Literacy Assessment Program: Foundation and Early Results

Abstract

The information literacy (IL) assessment program at Manhattan College in Riverdale, New York, instituted in 2014-2015, evaluates students' information literacy capabilities as demonstrated in their written coursework, their test performance, and their comments on library instruction sessions. Both instruction and assessment are closely linked to five learning objectives, and five years' assessment results have led to significant changes in the IL instruction program. This paper presents key concepts in IL assessment; highlights the importance of evidence-based measures (that is, direct assessment of cognitive outcomes); describes Manhattan College's three assessment methods, with guidelines for the reporting of results; discusses sampling difficulties and related statistical issues; describes the changes in IL instruction undertaken in response to the results ("closing the loop"); and reviews additional assessment methods that can help demonstrate the impact of IL instruction on broader educational outcomes. Context and Previous Work Key ConceptsAssessment is most useful when it is closely linked to teaching and learning. In the college or university setting, at least four stages of the learning and assessment cycle can be readily identified:

Evolution and diffusion of information literacy topics

“…Regarding the methods of evaluating information literacy, most scholars use only self-designed questionnaires that consist of self-assessments with closed-ended test questions (Pinto et al, 2019 ). Other scholars have combined interviews (Walters et al, 2020 ), experiments (Ding & Ma, 2013 ) and other survey methods to evaluate the developmental level of information literacy. However, these methods lack enthusiasm and flexibility, and the data collection process requires substantial cooperation from users, which is time-consuming and laborious.…”

“…With the development of the Internet, big data, artificial intelligence and other emerging technologies, massive and diverse process data can be recorded, which provides the possibility of data-driven evaluation. Various studies on technology behavior have used user process data; for example, Han et al ( 2019 ) assessed the degrees and features of teachers’ online participation in BL implementation, and Walters et al ( 2020 ) used students’ comments on a library instruction session to evaluate their information literacy. Kim et al ( 2020 ) used online search behavior to identify differences between self-perceived eHealth literacy and performance in judging the authenticity of cancer information.…”

Information Literacy Instruction in Asynchronous Online Courses: Which Approaches Work Best?

“…26 Walters asserts that "the distinction between evidence-based and perception-based measures is nearly identical to the distinction between cognitive and affective measures." 27 Cognitive, or evidence-based, measures look at data in which students demonstrate their skills as opposed to affective measures that examine students' confidence (for example) using those skills. Of indirect versus direct methods of data collection--which Walters says "are closely linked to cognitive or affective constructs" 28 --Walters explains that indirect methods, such as surveys or self-evaluations, in which students report what they do, build in a level of abstraction that introduces bias.…”

“…Of indirect versus direct methods of data collection--which Walters says "are closely linked to cognitive or affective constructs" 28 --Walters explains that indirect methods, such as surveys or self-evaluations, in which students report what they do, build in a level of abstraction that introduces bias. 29 Direct methods involve researchers looking "directly" at the data about which they are making claims (for example, looking at students' actual coursework rather than looking at what students say about having produced it). Camacho looked at how many students had watched an ILI video but did not measure how much students understood of what they had seen.…”

“…Summarize the three hypotheses proposed by the authors to explain why complex societies arose in Mesoamerica (hint: for a general discussion of social complexity, see Toby Evans, pp. [19][20][21][22][23][24][25][26][27][28]. How do the authors use the concept of "nuclear areas" in examining social complexity in Mesoamerica?…”