A framework for evaluating semantic metadata

Abstract

Because poor quality semantic metadata can destroy the effectiveness of semantic web technology by hampering applications from producing accurate results, it is important to have frameworks that support their evaluation. However, there is no such framework developed to date. In this context, we proposed i) an evaluation reference model, SemRef, which sketches some fundamental principles for evaluating semantic metadata, and ii) an evaluation framework, SemEval, which provides a set of instruments to support the detection of quality problems and the collection of quality metrics for these problems. A preliminary case study of SemEval shows encouraging results.

Quality assessment for Linked Data: A Survey

“…They explained that values are semantically accurate when they represent the correct state of an object. Based on this definition, we also considered the problems of spurious annotation and inaccurate annotation (inaccurate labeling and inaccurate classification) identified in Lei et al [40] related to the semantic accuracy dimension. The other articles [1,8,10,14,17,36,50,65] provide metrics for this dimension.…”

“…Redundancy occurs when there are equivalent schema elements with different names/identifiers (in case of intensional conciseness) and when there are equivalent 22 Under the assumption that we can infer that NewYork and London are different entities or, alternatively, make the unique name assumption. objects (instances) with different identifiers (in case of extensional conciseness) present in a dataset [40]. Kontokostas et al [36] provide metrics for this dimension.…”

“…of instances that violate the uniqueness rule total no. of relevant instances [19,36,40] -CN3: detection of unambiguous annotations/labels using the formula: 1 − no. of ambiguous instances no.…”

The Epistemology of Intelligent Semantic Web Systems

“…Indeed, these are often expressed in a way that makes them hard to formalize and test automatically [Lei et al, 2007]. Data quality evaluation is a difficult problem; simple indicators exist but they rarely capture the requirements of the applications.…”

On the Meaningfulness of “Big Data Quality” (Invited Paper)

“…Extensional conciseness measures the number of unique entities in relation to the overall number of entities in the data set [39]. Further, extensional conciseness can be measured as the total number of instances that violate the uniqueness rule in relation to the total number of relevant instances [20,35]. An example of intensional conciseness would be a particular flight, e.g., A123, being represented by two different properties in the same data set, such as http:// flights.org/airlineID and http://flights.org/name.…”