Aggregating pairwise semantic differences for few-shot claim verification

Zeng, Xia; Zubiaga, Arkaitz

doi:10.7717/peerj-cs.1137

Cited by 5 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…school science classrooms, and recently at the college level (Eden, 2023). LLMs have recently been used for factchecking and for identifying claim-evidence pairs in scienti c content (Koneru, Wu, and Rajtmajer, 2023;Wang et al, 2023;Zeng and Zubiaga, 2024). The hope is that LLMs could provide instructors and their students with assessment of the scienti c validity of student writing, aiming for the "gold standard" of conceptual learning (Gere et al, 2019).…”

Section: Declarationsmentioning

confidence: 99%

Using Large Language Models for Automated Grading of Student Writing about Science

Impey,

Wenger,

Garuda

et al. 2024

Preprint

View full text Add to dashboard Cite

A challenge in teaching large classes for formal or informal learners is assessing writing. As a result, most large classes, especially in science, use objective assessment tools like multiple choice quizzes. The rapid maturation of AI has created the possibility of using large language models (LLMs) to assess student writing. An experiment was carried out using GPT-3.5 and GPT-4 to see if machine learning methods based on LLMs can rival peer grading for reliability and automation in evaluating short writing assignments on topics in astronomy. The audience was lifelong learners in three massive open online courses (MOOCs) offered through Coursera. However, the results should also be applicable to non-science majors in university settings. The data was answers from 120 students on 12 questions across the three courses. The LLM was fed with total grades, model answers, and rubrics from an instructor for all three questions. In addition to seeing how reliably the LLMs reproduced instructor grades, the LLMs were asked to generate their own rubrics. Overall, the LLMs were more reliable than peer grading, both in the aggregate and by individual student, and they came much closer to the instructor grades for all three of the online courses. GPT-4 generally outperformed GPT-3.5. The implication is that LLMs can be used for automated, reliable, and scalable grading of student science writing.

show abstract

Section: Declarationsmentioning

confidence: 99%

Using Large Language Models for Automated Grading of Student Writing about Science

Impey,

Wenger,

Garuda

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…either supported or refuted. Zeng and Zubiaga (2023) explore active learning in combination with PET (Schick and Schütze, 2021), a popular prompt-based few-shot learning method, and Pan et al (2021) and Wright et al (2022) generate weakly supervised training data for zero-shot claim verification. However, none of the aforementioned methods produces (faithful) explanations.…”

Section: Related Workmentioning

confidence: 99%

QA-NatVer: Question Answering for Natural Logic-based Fact Verification

Aly,

Strong,

Vlachos

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Fact verification systems assess a claim's veracity based on evidence. An important consideration in designing them is faithfulness, i.e. generating explanations that accurately reflect the reasoning of the model. Recent works have focused on natural logic, which operates directly on natural language by capturing the semantic relation of spans between an aligned claim with its evidence via set-theoretic operators. However, these approaches rely on substantial resources for training, which are only available for high-resource languages. To this end, we propose to use question answering to predict natural logic operators, taking advantage of the generalization capabilities of instruction-tuned language models. Thus, we obviate the need for annotated training data while still relying on a deterministic inference system. In a few-shot setting on FEVER, our approach outperforms the best baseline by 4.3 accuracy points, including a state-of-the-art pre-trained seq2seq natural logic system, as well as a state-of-the-art prompt-based classifier. Our system demonstrates its robustness and portability, achieving competitive performance on a counterfactual dataset and surpassing all approaches without further annotation on a Danish verification dataset. A human evaluation indicates that our approach produces more plausible proofs with fewer erroneous natural logic operators than previous natural logic-based systems.

show abstract