SemTab 2019: Resources to Benchmark Tabular Data to Knowledge Graph Matching Systems

Abstract

Tabular data to Knowledge Graph matching is the process of assigning semantic tags from knowledge graphs (e.g., Wikidata or DBpedia) to the elements of a table. This task is a challenging problem for various reasons, including the lack of metadata (e.g., table and column names), the noisiness, heterogeneity, incompleteness and ambiguity in the data. The results of this task provide significant insights about potentially highly valuable tabular data, as recent works have shown, enabling a new family of data analytics and data science applications. Despite significant amount of work on various flavors of this problem, there is a lack of a common framework to conduct a systematic evaluation of state-of-the-art systems. The creation of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab) aims at filling this gap. In this paper, we report about the datasets, infrastructure and lessons learned from the first edition of the SemTab challenge.

Tough Tables: Carefully Evaluating Entity Linking for Tabular Data

“…• if the tables with columns sorted by type are annotated in the wrong way, then the entity linking algorithm is constrained by the type inferred looking at the first n rows, with n too small; • if homonyms or nicknames have been wrongly matched, that means that the algorithm employs popularity mechanisms (e.g., page rank), or it is based on a lookup service that returns the most popular entities first (e.g., DBpedia Lookup). 7 Annotating nicknames requires the algorithms to cover aspects of semantics that go a bit beyond simple heuristics; 8 • if the tables with level-1 noise are not properly annotated, then the algorithm cannot deal with real-world noise (that can be trickier than the artificial level-2 noise); • if the annotations are wrong for the tables containing nicknames, it might be the case the algorithm only focuses on the canonical names of the entities. Tables 1 and 2 show statistics for 2T and existing benchmark datasets.…”

“…It requires the introduction of semantic algorithms, namely semantic table interpreters, that link cells to elements in a KG. Recently, the SemTab 2019 [7] challenge was introduced to unify the community efforts towards the development of performing annotations. The challenge consists of different rounds in which tables of various difficulties have to be annotated.…”

“…In Figure 1, we show a case of ambiguity. In the Semantic Web community, this issue has already been highlighted, and it becomes necessary to have tables that resemble real use-cases [7]. In this paper, we propose a manually curated dataset for table annotation useful for evaluating specific aspects of the annotation task, and, in particular, of the entity linking task in table interpretation.…”

“…The authors themselves specify that the entity linking can be wrong sometimes, especially when the mention is ambiguous, spotlighting that there are a few ambiguous mentions in T2D. Recently, SemTab (Semantic Web Challenge on Tabular Data to Knowledge Graph Matching) was created to conduct a systematic evaluation of table annotation algorithms [7]. The organizers presented new datasets built by automatically generating tables from the results of SPARQL queries.…”

Neural Relation Extraction on Wikipedia Tables for Augmenting Knowledge Graphs

“…There is work on gathering and cleaning datasets of tables, amalgamating information from different tables, and returning tables as a result of keyword queries [1,17,[19][20][21]. Recent literature has introduced benchmarks and shared tasks in table understanding such as annotating columns or table cells with KG data [4,7]. However, relation extraction on has not been studied as much.…”