Automatic Identification of Nocuous Ambiguity

Abstract

We present the concept of nocuous ambiguity, which occurs when text is interpreted differently by different readers. In contrast, text exhibits innocuous ambiguity if different readers interpret it in the same way, even though structural or semantic analyses suggest that multiple interpretations may be possible. We collect multiple human judgements of a set of English phrases obtained from requirements documents. We focus on coordination ambiguity and show that across a group of judges there may be wide variation in what is perceived to be the correct interpretation. We develop the concept of an ambiguity threshold, which expresses the amount of variation between judgements that can be tolerated. We then develop and evaluate a heuristically based method of automatically predicting which sentences may be misunderstood for a given ambiguity threshold.

Automatic detection of nocuous coordination ambiguities in natural language requirements

“…Third, we employ the LogitBoost algorithm for the building of a so-called "nocuity classifier". We show that this machine learning algorithm performs better than the Logistic Regression algorithm that was used in our previous work [22]. Fourth, we implement an automated tool to detect and highlight on screen, nocuous ambiguity in text.…”

“…To select an appropriate machine learning (ML) algorithm to build our nocuity classifiers, we tested our dataset on a number of ML algorithms available in the WEKA package 9 including the logistic Regression algorithm that was used in our previous work [22]. Finally, we selected the LogitBoost algorithm for building the nocuity classifier, because it performed better than other candidates including decision trees, J48, Naive Bayes, SVM, and Logistic Regression.…”

“…In earlier work [25], we proposed a general methodology for automatic identification of nocuous ambiguity, which we use to guide our research on two types of ambiguity, coordination ambiguity [6,22] and anaphora ambiguity [24]. In contrast to other work, which is intended to resolve ambiguity [5,18], our research concerns identification of those ambiguities that are likely to lead to misunderstandings between different readers, while discounting those which tend to be interpreted in the same way by different readers despite their surface features.…”

“…Our previous work [6,22] focused on coordination ambiguity, a particularly common kind of structural ambiguity, highly prevalent in requirements documents. We investigated a methodology that used a number of heuristics based on corpus-based statistical information together with human judgments to predict whether a coordination ambiguity may be misunderstood for a given ambiguity threshold.…”

“…The work in this paper differs from our previous work [6,22] in the following four ways: first, we bring in information extraction techniques to tackle some problems in automatic detection of coordination ambiguity, which include pattern-based matching to detect ambiguous instances contained in the sentences, and named entity recognition (NER) techniques for the extraction of coordination constituents, such as coordinating conjuncts and the attached modifiers. Second, we introduce two more heuristics, collocation frequency in the local document and semantic similarity.…”