A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination

McIntosh, Timothy R.; Liu, Tong; Susnjak, Teo; Watters, Paul; Ng, Alex; Halgamuge, Malka N.

doi:10.1109/tai.2023.3332837

Cited by 48 publications

(35 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The evaluation of large language models has been rigorously conducted using a variety of benchmark datasets designed to test their capabilities across multiple dimensions [26,27]. Benchmarks like GLUE and SuperGLUE have set the standard for evaluating model performance on tasks such as text classification, sentiment analysis, and natural language inference, and they provide a comprehensive suite of tasks that collectively measure a model's ability to understand and generate human language accurately [26].…”

Section: Related Workmentioning

confidence: 99%

Modify Mistral Large Performance with Low-Rank Adaptation (LoRA) on the BIG-Bench Dataset

Fujiwara,

Kimura,

Nakano

2024

Preprint

View full text Add to dashboard Cite

Low-Rank Adaptation (LoRA) provides a resource-efficient method for fine-tuning large language models, addressing significant computational challenges. Implementing LoRA in the Mistral Large model significantly enhances its performance across a range of complex tasks. The evaluation conducted on the BIG-Bench dataset reveals substantial improvements in accuracy, precision, recall, and perplexity, highlighting the effectiveness of LoRA in optimizing large language models. By focusing on low-rank approximations, LoRA reduces the computational burden of fine-tuning, enabling efficient adaptation without compromising performance. The empirical results demonstrate that the LoRA-adapted model excels in mathematical problem-solving, creative text generation, and common sense reasoning, making it a practical and scalable solution for diverse AI applications. Additionally, the integration of LoRA facilitates seamless implementation with existing transformer architectures, ensuring compatibility and efficiency. The study emphasizes the transformative potential of LoRA to drive significant advancements in natural language processing, providing a resource-efficient method for model adaptation that ultimately enhances the capabilities and applicability of AI systems across various domains. By addressing the challenges of computational resources and adaptation efficiency, LoRA paves the way for more robust and versatile applications of large language models in real-world scenarios.

show abstract

Section: Related Workmentioning

confidence: 99%

Modify Mistral Large Performance with Low-Rank Adaptation (LoRA) on the BIG-Bench Dataset

Fujiwara,

Kimura,

Nakano

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Studies on the development of automated and scalable evaluation metrics for visual tasks have laid the groundwork for more reliable and comprehensive assessments of AI model performance in interpreting visual data. The creation of benchmark datasets and standardized testing environments aimed to mirror a range of real-world scenarios, thus providing a robust framework for measuring the accuracy and reliability of visual interpretations by AI systems [23], [24]. Innovative metrics designed to evaluate the qualitative aspects of visual outputs, such as the relevance and fidelity of generated images or the preciseness of image captions, marked a pivotal shift toward nuanced and context-aware evaluations [25], [26].…”

Section: Automated and Scalable Evaluation Metrics For Visual Tasksmentioning

confidence: 99%

Assessing Visual Hallucinations in Vision-Enabled Large Language Models

Lu,

Huang,

Wen

et al. 2024

Preprint

View full text Add to dashboard Cite

Recent advancements in vision-enabled large language models have prompted a renewed interest in evaluating their capabilities and limitations when interpreting complex visual data. The current research employs ImageNet-A, a dataset specifically designed with adversarially selected images that challenge standard AI models, to test the visual processing robustness of three prominent models: GPT-4 Vision, Google Gemini 1.5, and Anthropic Claude 3. Quantitative analyses revealed notable disparities in misclassification rates and types of errors among these models, indicating a variation in their ability to handle adversarial inputs effectively. GPT-4 Vision demonstrated a commendable robustness, whereas Google Gemini 1.5 excelled in processing speed and efficiency. Anthropic Claude 3, while showing intermediate accuracy levels, displayed a significant propensity for contextual misinterpretations. Qualitative evaluations further assessed the relevance and plausibility of the models' visual hallucinations, uncovering challenges in achieving human-like understanding of ambiguous or complex scenes. The findings emphasize the necessity for further improvements in semantic accuracy and contextual understanding. Future research directions include enhancing adversarial robustness, refining evaluation metrics to better capture the qualitative aspects of visual understanding, and fostering interdisciplinary collaborations to develop AI systems with more nuanced interpretive abilities. The study underscores the ongoing journey towards AI models that can match human perceptual skills, highlighting both the progress made and the considerable challenges that remain.

show abstract

“…The challenges in maintaining context and providing accurate references in responses were highlighted, revealing a gap in performance on tasks requiring deep knowledge or technical expertise [8], [10]. Another research article demonstrated the tendency of LLMs to generate plausible but factually incorrect information, underscoring the need for enhanced reasoning capabilities [11], [12]. The issue of data recency and the models' ability to incorporate the latest information was also discussed, emphasizing the static nature of LLM training data [13]- [15].…”

Section: A Limitations Of Llms In Information Retrieval and Reasoningmentioning

confidence: 99%

Merging Mixture of Experts and Retrieval Augmented Generation for Enhanced Information Retrieval and Reasoning

Xiong,

Zheng

2024

Preprint

View full text Add to dashboard Cite

This study investigates the integration of Retrieval Augmented Generation (RAG) into the Mistral 8x7B Large Language Model (LLM), which already uses Mixture of Experts (MoE), to address its existing limitations in complex information retrieval and reasoning tasks. By leveraging the Google BIG-Bench dataset, we conducted extensive quantitative and qualitative analyses to evaluate the augmented model's performance. The results demonstrate significant improvements in accuracy, precision, recall, and F1 score, highlighting the enhanced model's superior capability to generate contextually rich, accurate, and nuanced responses. This integration showcases a promising approach to overcoming the intrinsic limitations of traditional LLMs, marking a pivotal advancement in artificial intelligence research. Our findings contribute to the ongoing development of more adaptable, efficient, and intelligent AI systems, opening new avenues for AI application across various fields. The study acknowledges limitations related to dataset scope and computational demands, suggesting directions for future research to further refine and expand the model's applicability.

show abstract

A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination

Cited by 48 publications

References 41 publications

Modify Mistral Large Performance with Low-Rank Adaptation (LoRA) on the BIG-Bench Dataset

Modify Mistral Large Performance with Low-Rank Adaptation (LoRA) on the BIG-Bench Dataset

Assessing Visual Hallucinations in Vision-Enabled Large Language Models

Merging Mixture of Experts and Retrieval Augmented Generation for Enhanced Information Retrieval and Reasoning

Contact Info

Product

Resources

About