Academic librarians seeking to assess information literacy skills often focus on testing as a primary means of evaluation. Educators have long recognized the limitations of tests, and these limitations cause many educators to prefer rubric assessment to test-based approaches to evaluation. In contrast, many academic librarians are unfamiliar with the benefits of rubrics. Those librarians who have explored the use of information literacy rubrics have not taken a rigorous approach to methodology and interrater reliability. This article seeks to remedy these omissions by describing the benefits of a rubric-based approach to information literacy assessment, identifying a methodology for using rubrics to assess information literacy skills, and analyzing the interrater reliability of information literacy rubrics in the hands of university librarians, faculty, and students. Study results demonstrate that Cohen's κ can be effectively employed to check interrater reliability. The study also indicates that rubric training sessions improve interrater reliability among librarians, faculty, and students.
IntroductionAcademic librarians seeking to assess information literacy skills often focus on testing as a primary means of evaluation. Educators have long recognized the limitations of traditional tests that include fixed-choice question types (e.g., multiple choice, matching, and true/false). In fact, test limitations cause many K-16 educators to prefer rubric assessment to fixed-choice testing; however, most academic librarians have not adopted a rubric-based approach to information literacy assessment and therefore are unable to take advantage of the instructional benefits offered by rubric assessments. This article seeks to rectify this missed opportunity by describing the benefits of a rubric-based approach to information literacy assessment, identifying a methodology for using rubrics to assess information literacy skills, and analyzing the interrater reliability of information literacy rubrics in the hands of university librarians, faculty, and students.
Rubrics DefinedBased on assessment for learning (Oakleaf, 2009), motivation, and constructivist educational theory (Oakleaf, 2008, p. 244), rubrics are "descriptive scoring schemes" created by educators to guide analysis of student work (Moskal, 2000). Rubrics describe the parts and levels of performance of a particular task, product, or service (Hafner, 2003(Hafner, , p. 1509. Rubrics are often employed to judge quality (Popham, 2003, p. 95), and they can be used across a broad range of subjects (Moskal, 2000). Full-model rubrics, like the one used in this study, are formatted on a grid or table. They include criteria or target indicators down the left-hand side of the grid and list levels of performance across the top (Callison, 2000, p. 34). Criteria are the essential tasks or hallmarks that comprise a successful performance (Wiggins, 1996, p.V-6:2). Performance-level descriptors "spell out what is needed, with respect to each evaluative criterion . . . [for] ...