The ability to evaluate arguments in scientific texts: Measurement, cognitive processes, nomological network, and relevance for academic success at the university

Abstract

Background The evaluation of informal arguments is a key component of comprehending scientific texts and scientific literacy. Aim The present study examined the nomological network of university students’ ability to evaluate informal arguments in scientific texts and the relevance of this ability for academic success. Sample A sample of 225 university students from the social and educational sciences participated in the study. Methods Judgements of plausibility and the ability to recognize argumentation fallacies were assessed with a novel computer‐based diagnostic instrument (Argument Judgement Test; AJT). Results The items of the AJT partly conform to a 1‐PL model and test scores were systematically related to epistemological beliefs and verbal intelligence. Item‐by‐item analyses of responses and response times showed that implausible arguments were more difficult to process and correct responses to these items required increased cognitive effort. Finally, the AJT scores predicted academic success at university even if verbal intelligence and grade point average were controlled for. Conclusion These findings suggest that the ability to evaluate arguments in scientific texts is an aspect of rationality, relies on reflective processes, and is relevant for academic success.

The Role of Students’ Beliefs When Critically Reasoning From Multiple Contradictory Sources of Information in Performance Assessments

“…Particular investigation of argumentative skills would require a substantial change and further development of the experimental and assessment setting. For instance, there are several performance tests available that specifically focus on measuring argumentative skills (e.g., Argument Structure Test, Münchow et al, 2020 ; Agrument Judgment Test, Münchow et al, 2019 ) and are suitable for discriminant validation of CR assessments, which should be investigated in a follow-up research. In addition, comprehensive qualitative analyses of both the argumentative importance of the material on the one hand as well as (i) arguments (more or less reflective or intuitive, Mercier and Sperber, 2011 ) used by students in their responses and (ii) (new) arguments created by the students themselves based on given arguments in the provided information on the other hand need to be conducted in further studies, and explicitly linked to students’ critical reasoning ability and performance.…”

“…Students’ mental CR strategies may likely be insufficient for what is demanded for understanding heterogeneous information and, what is more, for effective and productive participation in a complex information environment (for a meta-study, see Huber and Kuncel, 2016 , for university students, see McGrew et al, 2018 ; Wineburg et al, 2018 ; Hahnel et al, 2019 ; Münchow et al, 2019 ; Shavelson et al, 2019 ; Zlatkin-Troitschanskaia et al, 2019b ). As a coping strategy, they may choose to reduce complexity by various means, for instance, by using cognitive heuristics, preferring simplified forms of information presentation, or relying on sources without verification, which can be exploited for manipulation ( Walthen and Burkell, 2002 ; Metzger, 2007 ; Horstmann et al, 2009 ; Metzger et al, 2010 ).…”

Assessing University Students' Critical Online Reasoning Ability: A Conceptual and Assessment Framework With Preliminary Evidence

“…Multiple-source comprehension assessment Lawless et al, 2012 Assessment of argument evaluation in scientific texts Münchow et al, 2019 Critical Thinking assessment in higher education Liu et al, 2014 Instructional Approaches (selection) Civic Online Reasoning instructional intervention McGrew et al, 2019 Critical source evaluation for improved search Leeder and Shah, 2016 Bad News game/multilingual browser game on use of major media disinformation strategies, based on 'inocculation' approach Roozenbeek and van der Linden, 2019 Review of CT interventions Abrami et al, 2008 Fostering CT using digital media/review of instructional designs in HE Jahn, 2012;Jahn and Kenner, 2018 CORA is based on approaches by Messick (1989) and Kane (2012) In Section Theoretical and Conceptual Framework, we first present the theoretical and conceptual COR framework, also in terms of related research approaches. In Section Assessment Framework of Critical Online Reasoning, we describe the U.S. assessment of civic online reasoning and present our work toward adapting and further developing this approach into an expanded assessment framework and scoring scheme for measuring COR in German higher education.…”

Untitled

“…С одной стороны, отечественные авторы в качестве предмета изучения выбирают традиционные виды способностей: пространственные (Аристова и др., 2018;Likhanov et al, 2018;Rodic et al, 2018), творческие (Ахметзянова, 2015; Богоявленская, Низовцова, 2017), мнемические (Murafa, 2015), математические (Богданова и др., 2019); с другой, предметом исследования становится изучение дискурсивных способностей (Воронин, 2018), понятийных (Kholodnaya, Emelin, 2015;Холодная и др., 2019), рефлексивных (По дойницина, 2017), способностей принятия решений (Федонкина, 2018) и ментализации (Холмогорова и др., 2015). Схожая ситуация характерна и для работ зарубежных психологов, в которых также осуществляется изучение традиционных видов способностей: речевых (Namaziandost et al, 2019), логических (Coppola et al, 2019), вербальных (Frick et al, 2019;Münchow et al, 2019), мнемических (Abellán-Mart nez et al, 2019), математических (Lauermann et al, 2020;Xie et al, 2020;Rodic et al, 2018), обучаемости (Kaplan et al, 2019) и чтения (Auphan et al, 2019;Kearns, Al Ghanem, 2019;Oerke et al, 2019). Однако и здесь появляются исследования, в которых традиционная классификация способностей расширяется за счет изучения способностей к саморегуляции (Miller, Bernacki, 2019), совладанию (Sun et al, 2018), эмоциональных (Mérida-López et al, 2019Mogil, 2019) и социальных (Mogil, 2019) способностей.…”