User-driven quality evaluation of DBpedia

Abstract

Linked Open Data (LOD) comprises of an unprecedented volume of structured datasets on the Web. However, these datasets are of varying quality ranging from extensively curated datasets to crowdsourced and even extracted data of relatively low quality. We present a methodology for assessing the quality of linked data resources, which comprises of a manual and a semi-automatic process. The first phase includes the detection of common quality problems and their representation in a quality problem taxonomy. In the manual process, the second phase comprises of the evaluation of a large number of individual resources, according to the quality problem taxonomy via crowdsourcing. This process is accompanied by a tool wherein a user assesses an individual resource and evaluates each fact for correctness. The semi-automatic process involves the generation and verification of schema axioms. We report the results obtained by applying this methodology to DBpedia. We identified 17 data quality problem types and 58 users assessed a total of 521 resources. Overall, 11.93% of the evaluated DBpedia triples were identified to have some quality issues. Apply- * Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. ISEM '13, September ing the semi-automatic component yielded a total of 222,982 triples that have a high probability to be incorrect. In particular, we found that problems such as object values being incorrectly extracted, irrelevant extraction of information and broken links were the most recurring quality problems. With this study, we not only aim to assess the quality of this sample of DBpedia resources but also adopt an agile methodology to improve the quality in future versions by regularly providing feedback to the DBpedia maintainers.

Sustainable Linked Data Generation: The Case of DBpedia

“…Domain validation of the mapping templates defined in the custom dbpedia mapping language can not be supported [6]. Nevertheless, the dbpedia dataset quality would significantly improve if schema violations is applied to the mapping templates [6,13,19]. Currently custom dbpedia mapping templates are only validated for syntax violations (using the Mapping Syntax Validator 15 [12]).…”

“…Refining and extending the mapping rules required several iterations before we reach to a version that generates the same rdf triples as the current dbpedia ef. Moreover, even though it is convenient that certain rdf statements are generated without the community being involved, the complete lack of control over what is generated or how the data values are transformed, it is often the cause for deteriorating dbpedia dataset quality [18,19]. Then again, declaratively defining everything causes an overhead when editing the mapping rules.…”

“…Even though dbpedia is one of the central interlinking hubs in the Linked Open Data (lod) cloud [17], its generation framework has limitations that reflect on the generated dataset [14,19]. We distinguish two types of issues with dbpedia: schema-level rules that define how to apply vocabularies to raw data [6,10,14], e.g., the dbo:militaryBranch property is used for entities of dbo:MilitaryUnit type, but it should only be used with entities of dbo:Person type [13].…”

“…data-level extracted, or processed and transformed data values. The former includes incorrect, incomplete or irrelevant extracted values and datatypes, or not (well-)recognized templates [19]; the latter issues with parsing values [18], interpreting/converting units [18], or transforming cardinal direction [15].…”

“…Errors or inconsistencies in dbpedia can be addressed by amending the dbpedia ef, mapping rules, or Wikipedia itself [19]. However, even though several changes were applied to both the dbpedia ef and mapping rules [12], quality issues still persist.…”