A Testing Framework for AI Linguistic Systems (testFAILS)

Abstract

This paper presents an innovative testing framework, testFAILS, designed for the rigorous evaluation of AI Linguistic Systems (AILS), with particular emphasis on the various iterations of ChatGPT. Leveraging orthogonal array coverage, this framework provides a robust mechanism for assessing AI systems, addressing the critical question, “How should AI be evaluated?” While the Turing test has traditionally been the benchmark for AI evaluation, it is argued that current, publicly available chatbots, despite their rapid advancements, have yet to meet this standard. However, the pace of progress suggests that achieving Turing-test-level performance may be imminent. In the interim, the need for effective AI evaluation and testing methodologies remains paramount. Ongoing research has already validated several versions of ChatGPT, and comprehensive testing on the latest models, including ChatGPT-4, Bard, Bing Bot, and the LLaMA and PaLM 2 models, is currently being conducted. The testFAILS framework is designed to be adaptable, ready to evaluate new chatbot versions as they are released. Additionally, available chatbot APIs have been tested and applications have been developed, one of them being AIDoctor, presented in this paper, which utilizes the ChatGPT-4 model and Microsoft Azure AI technologies.

Robust Testing of AI Language Models Resilience with Novel Adversarial Prompts

“…Future research will extend beyond natural language processing, focusing on the resilience of state-of-the-art LLMs against multimodal adversarial attacks involving both text and images. The objectives will include evaluating the vulnerability of LLMs to combined text and image-based adversarial attacks and proposing novel strategies to enhance the resilience of multimodal AI systems [36][37][38]. This direction aligns with the evolving landscape of AI, where understanding and countering sophisticated adversarial tactics become increasingly vital.…”

Robust Testing of AI Language Model Resiliency with Novel Adversarial Prompts

“…This new direction aims to assess the vulnerability of LLMs to more sophisticated and composite adversarial strategies, thereby contributing to the development of more robust and resilient AI systems. The objectives will include evaluating the vulnerability of LLMs to combined text-and image-based adversarial attacks and proposing novel strategies to enhance the resilience of multimodal AI systems [39][40][41].…”

ChatGPT Translation of Program Code for Image Sketch Abstraction

“…In the context of scientific coding, Cory Merow et al [10] found that AI chatbots could boost scientific coding by assisting scientists in writing code more quickly and accurately. Yulia Kumar et al [11] proposed a comprehensive testing framework for AI linguistic systems using ChatGPT (version 4) as an AI pair programmer. They found that ChatGPT could generate test cases more effectively than human experts, indicating its potential in the M-to-PY conversion for complex projects such as image skeletonization.…”