Natural Test Generation for Precise Testing of Question Answering Software

Shen, Qingchao; Chen, Junjie; Zhang, Jie M.; Wang, Haoyu; Liu, Shuang; Tian, Menghan

doi:10.1145/3551349.3556953

Cited by 20 publications

(5 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach benefits from existing techniques for evaluating question-answer (QA) language models [12,30,59,78]. Multiple studies proposed reliability testing techniques for the QA software [11,20,34,47,59,60]. Among them, Shen et al [60] outperformed previous state-of-the-art approaches by discovering 23% more bugs or inconsistencies in the target answer-generating models and the Challenger component adopted their question mutation technique for generating challenging questions.…”

Section: Mutationmentioning

confidence: 99%

“…Multiple studies proposed reliability testing techniques for the QA software [11,20,34,47,59,60]. Among them, Shen et al [60] outperformed previous state-of-the-art approaches by discovering 23% more bugs or inconsistencies in the target answer-generating models and the Challenger component adopted their question mutation technique for generating challenging questions. For this, our mutation challenger employs the sentence-level metamorphic testing technique, QAQA, proposed by [60].…”

Section: Mutationmentioning

confidence: 99%

“…Among them, Shen et al [60] outperformed previous state-of-the-art approaches by discovering 23% more bugs or inconsistencies in the target answer-generating models and the Challenger component adopted their question mutation technique for generating challenging questions. For this, our mutation challenger employs the sentence-level metamorphic testing technique, QAQA, proposed by [60]. It inserts a redundant sentence as a clause to the original (basic challenge) question to generate the mutated question which puts a larger cognitive load on ChatGPT and challenges it further to reveal inconsistency with the responses (𝑅 𝐶𝑀 ).…”

Section: Mutationmentioning

confidence: 99%

See 2 more Smart Citations

ChatGPT Incorrectness Detection in Software Reviews

Tanzil,

Khan,

Uddin

2024

Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

View full text Add to dashboard Cite

We conducted a survey of 135 software engineering (SE) practitioners to understand how they use Generative AI-based chatbots like ChatGPT for SE tasks. We find that they want to use ChatGPT for SE tasks like software library selection but often worry about the truthfulness of ChatGPT responses. We developed a suite of techniques and a tool called CID (ChatGPT Incorrectness Detector) to automatically test and detect the incorrectness in ChatGPT responses. CID is based on the iterative prompting to ChatGPT by asking it contextually similar but textually divergent questions (using an approach that utilizes metamorphic relationships in texts). The underlying principle in CID is that for a given question, a response that is different from other responses (across multiple incarnations of the question) is likely an incorrect response. In a benchmark study of library selection, we show that CID can detect incorrect responses from ChatGPT with an F1-score of 0.74 -0.75. CCS CONCEPTS• Computing methodologies → Natural language processing; • Software and its engineering → Software testing and debugging.

show abstract

Section: Mutationmentioning

confidence: 99%

Section: Mutationmentioning

confidence: 99%

Section: Mutationmentioning

confidence: 99%

See 1 more Smart Citation

ChatGPT Incorrectness Detection in Software Reviews

Tanzil,

Khan,

Uddin

2024

Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

View full text Add to dashboard Cite

show abstract

“…Shen et al 67 leveraged sentence‐level mutation to generate natural test inputs to achieve precise testing of question answering software. You et al 68 proposed DRFuzz, which aims at generating test inputs triggering regression faults for DNN models.…”

Section: Related Workmentioning

confidence: 99%

Revisiting deep neural network test coverage from the test effectiveness perspective

Yan

Chen

Cao

et al. 2023

J Software Evolu Process

Self Cite

View full text Add to dashboard Cite

Many test coverage metrics have been proposed to measure the deep neural network (DNN) testing effectiveness, including structural coverage and nonstructural coverage. These test coverage metrics are proposed based on the fundamental assumption: They are correlated with test effectiveness. However, the fundamental assumption is still not validated sufficiently and reasonably, which brings question on the usefulness of DNN test coverage. This paper conducted a revisiting study on the existing DNN test coverage from the test effectiveness perspective, to effectively validate the fundamental assumption. Here, we carefully considered the diversity of subjects, three test effectiveness criteria, and both typical and state‐of‐the‐art test coverage metrics. Different from all the existing studies that deliver negative conclusions on the usefulness of existing DNN test coverage, we identified some positive conclusions on their usefulness from the test effectiveness perspective. In particular, we found the complementary relationship between structural and nonstructural coverage and identified the practical usage scenarios and promising research directions for these existing test coverage metrics.

show abstract

“…NLP software, such as machine translation software and chatbot, has also been widely used in human life. Similar to AI software, researchers have proposed variance methods to validate the reliability of NLP software on the correctness [18,19,40,44], toxicity [53,54], fairness [47,52,57].…”

Section: Related Work 61 Testing Of Ai Softwarementioning

confidence: 99%

Characterization of Two BAHD Acetyltransferases Highly Expressed in the Flowers of Jasminum sambac (L.) Aiton

Wang

Zhang

Chen

et al. 2021

Plants

View full text Add to dashboard Cite

Volatile benzenoid compounds are found in diverse aromatic bouquets emitted by most moth-pollinated flowers. The night-blooming Jasminum sambac is widely cultivated worldwide in the tropics and subtropics for ornamental and industrial purposes owing to its fragrant flowers. Benzylacetate is a characteristic constituent in jasmine scent which makes up to approximately 20–30% of the total emission in the headspace or extract, but the biosynthesis enzymes and the encoding genes have not yet been described. Here, we identify two cytosolic BAHD acyltransferases specifically expressed in the petals with a positive correlation closely to the emission pattern of the volatile benzenoids. Both JsBEAT1 and JsBEAT2 could use benzylalcohol and acetate-CoA as substrates to make benzylacetate in vitro. The recombinant GST-JsBEAT1 has an estimated apparent Km of 447.3 μM for benzylalcohol and 546.0 μM for acetate-CoA, whereas in the instance of the His-JsBEAT2, the Km values are marginally lower, being 278.7 and 317.3 μM, respectively. However, the catalytic reactions by the GST-JsBEAT1 are more efficient than that by the His-JsBEAT2, based on the steady-state kcat parameters. Furthermore, ectopic expression of JsBEAT1 and JsBEAT2 in the transgenic P. hybrida plants, driven by a flower-specific promotor, significantly enhances the biosynthesis of benzylbenzoate and benzylacetate, as well as the total VOCs.

show abstract

Natural Test Generation for Precise Testing of Question Answering Software

Cited by 20 publications

References 26 publications

ChatGPT Incorrectness Detection in Software Reviews

ChatGPT Incorrectness Detection in Software Reviews

Revisiting deep neural network test coverage from the test effectiveness perspective

Characterization of Two BAHD Acetyltransferases Highly Expressed in the Flowers of Jasminum sambac (L.) Aiton

Contact Info

Product

Resources

About