Assessing University Students' Critical Online Reasoning Ability: A Conceptual and Assessment Framework With Preliminary Evidence

Abstract

Critical evaluation skills when using online information are considered important in many research and education frameworks; critical thinking and information literacy are cited as key twenty-first century skills for students. Higher education may play a special role in promoting students' skills in critically evaluating (online) sources. Today, higher education students are more likely to use the Internet instead of offline sources such as textbooks when studying for exams. However, far from being a value-neutral, curated learning environment, the Internet poses various challenges, including a large amount of incomplete, contradictory, erroneous, and biased information. With low barriers to online publication, the responsibility to access, select, process, and use suitable relevant and trustworthy information rests with the (self-directed) learner. Despite the central importance of critically evaluating online information, its assessment in higher education is still an emerging field. In this paper, we present a newly developed theoretical-conceptual framework for Critical Online Reasoning (COR), situated in relation to prior approaches (“information problem-solving,” “multiple-source comprehension,” “web credibility,” “informal argumentation,” “critical thinking”), along with an evidence-centered assessment framework and its preliminary validation. In 2016, the Stanford History Education Group developed and validated the assessment of Civic Online Reasoning for the United States. At the college level, this assessment holistically measures students' web searches and evaluation of online information using open Internet searches and real websites. Our initial adaptation and validation indicated a need to further develop the construct and assessment framework for evaluating higher education students in Germany across disciplines over their course of studies. Based on our literature review and prior analyses, we classified COR abilities into three uniquely combined facets: (i) online information acquisition, (ii) critical information evaluation, and (iii) reasoning based on evidence, argumentation, and synthesis. We modeled COR ability from a behavior, content, process, and development perspective, specifying scoring rubrics in an evidence-centered design. Preliminary validation results from expert interviews and content analysis indicated that the assessment covers typical online media and challenges for higher education students in Germany and contains cues to tap modeled COR abilities. We close with a discussion of ongoing research and potentials for future development.

Performance Assessment and Digital Training Framework for Young Professionals' Generic and Domain-Specific Online Reasoning in Law, Medicine, and Teacher Practice

“…Our research in the collaborative BRIDGE project, funded by the German Federal Ministry of Education and Research (BMBF) and conducted under the umbrella of the newly established federal program "Research for the Design of Educational Processes under the Conditions of Digital Change," 1 is based on our previous work on the assessment of generic skills in higher education in the international projects CLA+ (Zlatkin-Troitschanskaia et al, 2018), iPAL (Shavelson et al, 2019), and CORA (Molerov et al, 2020) as well as on experiences with jobspecific performance assessments from the research programs KoKoHs (Zlatkin-Troitschanskaia et al, 2020) and ASCOT+ (BMBF, 2015), which measured professional competence.…”

“…Overall, no studies on the use of online information among young professionals in law and among medical interns were found. However, the preliminary research (Nagel et al, 2020a;Molerov et al, 2020;Schmidt et al, 2020;Banerjee et al, 2020) indicates that COR is of great importance for successful learning in digital contexts and that its facets (as described in Section 2.1), including the ability to critically evaluate online information, can be developed and trained in a targeted manner.…”

“…To allow this American assessment to be used in a German context as well, we adapted it for tertiary education in Germany (for details, see Molerov et al, 2020). Due to (socio)cultural embeddedness of US-American items and the use of real websites, mere translation was not sufficient, but a functionally equivalent redesign of the assessment proved feasible (though several limitations emerged with regard to creating a functional German equivalent, (see Molerov et al, 2020).…”

“…To allow this American assessment to be used in a German context as well, we adapted it for tertiary education in Germany (for details, see Molerov et al, 2020). Due to (socio)cultural embeddedness of US-American items and the use of real websites, mere translation was not sufficient, but a functionally equivalent redesign of the assessment proved feasible (though several limitations emerged with regard to creating a functional German equivalent, (see Molerov et al, 2020). During specification of the key construct criteria for redesign, we also further generalized the focus to encompass not only civic issues, but topics related to students' studies in higher education for which they may acquire information online (e.g., from colleagues within the profession), and additionally, further specified the facets (for details, Molerov et al, 2020).…”

“…Due to (socio)cultural embeddedness of US-American items and the use of real websites, mere translation was not sufficient, but a functionally equivalent redesign of the assessment proved feasible (though several limitations emerged with regard to creating a functional German equivalent, (see Molerov et al, 2020). During specification of the key construct criteria for redesign, we also further generalized the focus to encompass not only civic issues, but topics related to students' studies in higher education for which they may acquire information online (e.g., from colleagues within the profession), and additionally, further specified the facets (for details, Molerov et al, 2020). Based on expert interviews which particularly stressed the Humboldts' tradition of critical thinking in German higher education (Beck, 2020), as well as on a systematic synthesis of relevant and related multifaceted constructs, such as critical thinking (Facione, 1990;Jahn & Kenner, 2018), digital literacy (Ainley et al, 2016), media literacy (Abreu et al 2017), information literacy (Taylor & Dalal, 2014), critical argumentation (Walton, 2006a), scientific reasoning and argumentation (Fischer et al, 2018) and information problem-solving (Brand-Gruwel et al, 2017), we developed a detailed conceptual framework titled critical online reasoning -a crucial skill in competently using online information (COR; for further discussion of related constructs see, Molerov et al, 2020).…”