The effectiveness of argumentation in tutorial dialogues with an Intelligent Tutoring System for genetic risk of breast cancer

Abstract

BRCA Gist is an Intelligent Tutoring System that helps women understand issues related to genetic testing and breast cancer risk. In two laboratory experiments and a field experiment with community and web-based samples, an avatar asked 120 participants to produce arguments for and against genetic testing for breast cancer risk. Two raters assessed the number of argumentation elements (claim, reason, backing, etc.) found in response to prompts soliciting arguments for and against genetic testing for breast cancer risk (IRR=.85). When asked to argue for genetic testing, 53.3 % failed to meet the minimum operational definition of making an argument, a claim supported by one or more reasons. When asked to argue against genetic testing, 59.3 % failed to do so. Of those who failed to generate arguments most simply listed disconnected reasons. However, participants who provided arguments against testing (40.7 %) performed significantly higher on a posttest of declarative knowledge. In each study we found positive correlations between the quality of arguments against genetic testing (i.e., number of argumentation elements) and genetic risk categorization scores. Although most interactions did not contain two or more argument elements, when more elements of arguments were included in the argument against genetic testing interaction, participants had greater learning outcomes. Apparently, many participants lack skills in making coherent arguments. These results suggest an association between argumentation ability (knowing how to make complex arguments) and subsequent learning. Better education in developing arguments may be necessary for people to learn from generating arguments within Intelligent Tutoring Systems and other settings.

Understanding genetic breast cancer risk: Processing loci of the BRCA Gist Intelligent Tutoring System

“…Replicating previous findings, (Cedillos-Whynott et al, in press) fine-grained analyses indicate that a majority of participants simply listed reasons – which fails to meet the minimum operational definition of an argument (Wolfe, Britt & Butler, 2009). However, when participants actually engaged in argumentation, they showed gains in knowledge and comprehension partially supporting Hypothesis 7.…”

“…Overall, only 46.5% of these dialogues met the minimum criteria for being an argument, operationalized as a claim supported by one or more reasons with the connecting warrant implied but not necessarily stated. This is comparable to the 43.7% found by Cedillos-Whynott and colleagues (in press). Overall, 1% earned a 0 meaning no reasons provided; 52.5% earned a 1 meaning reasons are stated or listed without any connection to a claim; 18.5% earned a 2 meaning claims and reasons are stated and the warrant is implied but not stated; 21% earner a 3 meaning three or more argument elements, (claim, reason, backing, counterargument, rebuttal etc.…”

“…With respect to Hypothesis 7, a comparison of results from the current study to previous research on BRCA Gist conducted by Cedillos-Whynott and colleagues (in press) reveals some striking similarities. We found a comparable pattern for the percentage of rubric items covered in the two argumentation dialogues as predictors of outcomes.…”

“…Following the procedure of Cedillos-Whynott and colleagues (in press) we subjected each argumentation dialogue to analysis with a rubric assessing whether each dialogue met the criteria for being an argument and the degree of sophistication in using elements of argumentation. Overall, only 46.5% of these dialogues met the minimum criteria for being an argument, operationalized as a claim supported by one or more reasons with the connecting warrant implied but not necessarily stated.…”

“…We subjected each of the argumentation dialogues to an analysis using rubrics assessing the presence or absence of elements or argumentation (Cedillos-Whynott, Wolfe, Widmer, Brust-Renck, Weil, & Reyna, in press; Wolfe, Britt & Butler, 2009). Each argumentation dialogue was assigned a score from 0 to 4 where 0 = no reasons provided; 1 = reasons are listed or stated without any connection to a claim; 2 = claims and reasons are stated and the connection between them (the warrant) is implied but not stated; 3 = three or more argument elements – in addition to the warrant – are implied but not stated (examples include claim, reason, backing, counterargument, rebuttal); and 4 = three or more argument elements, in addition to the warrant, are explicitly stated.…”