A Framework for the Evaluation of SUMO-Based Ontologies Using WordNet

Abstract

This paper offers a new practical approach toward automated commonsense reasoning with first-order logic (FOL) SUMO-based ontologies. We propose a new black-box evaluation framework for SUMO-based ontologies, which exploits the world knowledge encoded in WordNet and its mapping into SUMO. Our proposal consists of both a novel semi-automatic method for the creation of a large set of commonsense problems and a new procedure that enables its automatic evaluation by using automated theorem provers (ATPs). The application of our method enables the creation of a very large benchmark consisting of more than 15 000 problems from a small set of manually built question patterns that exploit the WordNet semantic relations. By means of the resulting benchmark, we successfully evaluate the competency of different translations of SUMO into FOL and the performance of various state-of-the-art FOL ATPs according to several quality criteria. A general analysis of our experimental results demonstrates that the proposed commonsense problems are heterogeneous and non-trivial. Furthermore, a fine-grained analysis of the experimental results obtained for a sample of our benchmark enables the detection of some mapping errors and some discrepancies between the knowledge of WordNet and SUMO. The evaluation benchmark and all the resources that have been used and developed during this work are released in a single package. INDEX TERMS Ontology evaluation, automated reasoning, commonsense knowledge, WordNet, SUMO.

Towards Word Sense Disambiguation by Reasoning

“…Further, the fact that all the CQs resulting from the QP event remain unsolved is also not surprising since the lack of knowledge about events in the ontology was also detected in the experimentation reported in [9] and confirmed in [6]. On the contrary, the large amount of solved CQs belong to the QPs hypernymy and cohyponymy as expected, because the best results in the experimentations reported in [9,10] are obtained for the CQs based on hypernymy/hyponymy.…”

“…On one hand, we are improving the involved knowledge resources in several ways. First, we are correcting mapping and knowledge errors by applying both black- [10] and white-box testing [7] techniques. Second, we are improving the quality of the mapping between WordNet and SUMO by focusing on particular relations such as meronymy [4,5] and metonymy [16].…”

“…WordNet and SUMO (and therefore Adimen-SUMO) are connected in a semantic mapping [21] by means of three semantic relations: equivalence, subsumption and instance. By exploiting this mapping, we apply the black-box testing method described in [10] that enables the automatic creation a set of conjectures for each word pair by considering the semantic relations provided by BLESS. Then, these conjectures are evaluated using Vampire and, according to the results, each word is disambiguated to a single synset.…”

“…To this end, we conveniently use of our method for the evaluation of SUMO-based ontologies [10]. This method is an adaptation of the methodology for the design and evaluation of ontologies introduced in [17], which is based on the use of competency questions (CQs): problems that an ontology is expected to answer.…”