Towards ontology evaluation across the life cycle

Abstract

Executive summary ProblemCurrently, there is no agreed on methodology for development of ontologies, and there is no consensus on how ontologies should be evaluated. Consequently, evaluation techniques and tools are not widely utilized in the development of ontologies. This can lead to ontologies of poor quality and is an obstacle to the successful deployment of ontologies as a technology. ApproachThe goal of the Ontology Summit 2013 was to create guidance for ontology developers and users on how to evaluate ontologies. Over a period of four months a variety of approaches were discussed by participants, who represented a broad spectrum of ontology, software, and system developers and users. We explored how established best practices in systems engineering and in software engineering can be utilized in ontology development. ResultsThis document focuses on the evaluation of five aspects of the quality of ontologies: intelligibility, fidelity, craftsmanship, fitness, and deployability. A model for the ontology life cycle is presented, and F. Neuhaus et al. / Communiquéevaluation criteria are presented in the context of the phases of the life cycle. We discuss the availability of tools and the document ends with observations and recommendations. Given the current level of maturity of ontology as an engineering discipline, any results on how to best build and evaluate ontologies have to be considered as preliminary. However, the results achieved a broad consensus across the range of backgrounds, application foci, specialties and experience found in the Ontology Summit community. RecommendationsFor more reliable success in ontology development and use, ontology evaluation should be incorporated across all phases of the ontology life cycle. Evaluation should be conducted against carefully identified requirements; these requirements depend on the intended use of the ontology and its operational environment. For this reason, we recommend the development of integrated ontology development and management environments that support the tracking of requirements for, and the evaluation of, ontologies across all phases of their development and use. Purpose of this documentThe purpose of this document is to advance the understanding and adoption of ontology evaluation practices. Our focus is on the critical relationships between usage requirements, the life cycle of an ontology, evaluation, and the quality of the result.This document is rooted in the 2013 Ontology Summit. Over four months, Summit participants prepared and presented materials, shared references, suggested resources, discussed issues and materials by email list, and met virtually each week for presentations and discussions. This Summit had the focal topic "Ontology Evaluation across the Ontology Lifecycle". This document represents a synthesis of a subset of ideas presented, discussed, and developed over the course of these four months, and reflects the contributions of the Summit's participants and the consensus of the Summit community.The intended audience for th...

Toward a standard ontology of surgical process models

“…), such tools are still lacking for ontology engineering. Several ontology development best-practices and methodologies exist [52,72,77,78], but none of them have been accepted at a large scale. As ontology development is in its essence a collaborative work, the lack of mature tools for their collaborative development is a major obstacle.…”

Bootstrapping the applied ontology practice: Ontology communities, then and now

“…• Ontology Summit 2013 -"Ontology Evaluation Across the Ontology Lifecycle" 16 (Neuhaus et al, 2013). • Ontology Summit 2014 -"Big Data and Semantic Web Meet Applied Ontology" 17 (Obrst et al, 2014).…”

Cohesion Metrics for Evaluating Semantic Web Ontologies

“…According to the measurement results of the ontologies, developers can reconstruct ontologies, and users can select the optimal structure of the ontologies between different systems. For more reliable success in ontology development and use, ontology evaluation should be incorporated across all phases of the ontology life cycle [2].…”