LogiQA 2.0—An Improved Dataset for Logical Reasoning in Natural Language Understanding

Liu, Hanmeng; Liu, J.; Cui, Leyang; Teng, Zhiyang; Duan, Nan; Zhou, Ming; Zhang, Yue

doi:10.1109/taslp.2023.3293046

Cited by 9 publications

(10 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One plausible explanation for this pattern could be that the summaries were also generated by the same model. As previously demonstrated, GPTbased models tend to favor their own generated text more than text generated by other models (Chiang and Liu et al, 2023a). Conversely, these models exhibit a high degree of sensitivity to the provided prompt and input data.…”

Section: Comparing Human and Gpt-4 As Evaluatorsmentioning

confidence: 87%

“…This pattern indicated a low level of agreement with human annotators. Therefore, we could not reproduce the same results as Liu et al (2023a).…”

Section: Comparing Human and Gpt-4 As Evaluatorsmentioning

confidence: 92%

“…To ensure the validity and effectiveness of the generated TikTok content, an empirical study was conducted following the methodology proposed by Liu et al (2023a). For the evaluation process, five participants were recruited from Amazon Mechanical Turk (AMT).…”

Section: Comparing Human and Gpt-4 As Evaluatorsmentioning

confidence: 99%

“…Given the high cost of human evaluation, we opted to assess the summaries using an evaluative template prompt created for GPT-4, following the same instructions as provided to human participants, described in Figure Number. We evaluated the same 150 samples with only the GPT-4 model following Liu et al (2023a), and focused on this model as earlier versions did not demonstrate the level of performance achieved by this one.…”

Section: Comparing Human and Gpt-4 As Evaluatorsmentioning

confidence: 99%

See 3 more Smart Citations

Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems

2023

View full text Add to dashboard Cite

The continuous progress in Named Entity Recognition allows the identification of complex entities in multiple domains. The traditionally used metrics like precision, recall, and F1-score can only reflect the classification quality of the underlying NER model to a limited extent. Existing metrics do not distinguish between a non-recognition of an entity and a misclassification of an entity. Additionally, the dealing with redundant entities remains unaddressed. We propose WRF, a Weighted Rouge F1 metric for Entity Recognition, to solve the mentioned gaps in currently available metrics. We successfully employ the WRF metric for automotive entity recognition, followed by a comprehensive qualitative and quantitative analysis of the obtained results.

show abstract

Section: Comparing Human and Gpt-4 As Evaluatorsmentioning

confidence: 87%

“…This pattern indicated a low level of agreement with human annotators. Therefore, we could not reproduce the same results as Liu et al (2023a).…”

Section: Comparing Human and Gpt-4 As Evaluatorsmentioning

confidence: 92%

Section: Comparing Human and Gpt-4 As Evaluatorsmentioning

confidence: 99%

Section: Comparing Human and Gpt-4 As Evaluatorsmentioning

confidence: 99%

See 2 more Smart Citations

Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems

2023

View full text Add to dashboard Cite

show abstract

“…Logical Reasoning For logical reasoning data resources, they can be mainly divided into two categories: Natural Language Inference (Saha et al, 2020;Tian et al, 2021;Liu et al, 2021) and Multiple-Choice Reading Comprehension (Liu et al, 2020Liu et al, 2023a). So far, there have been studies that have conducted in-depth analyses of the performance of LLMs on these two types of tasks.…”

Section: Reasoning Evaluation Of Llmsmentioning

confidence: 99%

Building Chinese interdisciplinary research centers : the case of Tsinghua University

Li¹,

李蓉暉²

View full text Add to dashboard Cite

Graph machine learning has been extensively studied in both academia and industry. However, in the literature, most existing graph machine learning models are designed to conduct training with data samples in a random order, which may suffer from suboptimal performance due to ignoring the importance of different graph data samples and their training orders for the model optimization status. To tackle this critical problem, curriculum graph machine learning (Graph CL), which integrates the strength of graph machine learning and curriculum learning, arises and attracts an increasing amount of attention from the research community. Therefore, in this paper, we comprehensively overview approaches on Graph CL and present a detailed survey of recent advances in this direction. Specifically, we first discuss the key challenges of Graph CL and provide its formal problem definition. Then, we categorize and summarize existing methods into three classes based on three kinds of graph machine learning tasks, i.e., node-level, linklevel, and graph-level tasks. Finally, we share our thoughts on future research directions. To the best of our knowledge, this paper is the first survey for curriculum graph machine learning.

show abstract

MLRQA: A Dataset with Multimodal Logical Reasoning Challenges

Xiao,

Lin,

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

LogiQA 2.0—An Improved Dataset for Logical Reasoning in Natural Language Understanding

Cited by 9 publications

References 66 publications

Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems

Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems

Building Chinese interdisciplinary research centers : the case of Tsinghua University

MLRQA: A Dataset with Multimodal Logical Reasoning Challenges

Contact Info

Product

Resources

About