Schema-Agnostic Progressive Entity Resolution

Abstract

Entity Resolution (ER) is the task of finding entity profiles that correspond to the same real-world entity. Progressive ER aims to efficiently resolve large datasets when limited time and/or computational resources are available. In practice, its goal is to provide the best possible partial solution by approximating the optimal comparison order of the entity profiles. So far, Progressive ER has only been examined in the context of structured (relational) data sources, as the existing methods rely on schema knowledge to save unnecessary comparisons: they restrict their search space to similar entities with the help of schema-based blocking keys (i.e., signatures that represent the entity profiles). As a result, these solutions are not applicable in Big Data integration applications, which involve large and heterogeneous datasets, such as relational and RDF databases, JSON files, Web corpus etc. To cover this gap, we propose a family of schema-agnostic Progressive ER methods, which do not require schema information, thus applying to heterogeneous data sources of any schema variety. First, we introduce two naïve schema-agnostic methods, showing that straightforward solutions exhibit a poor performance that does not scale well to large volumes of data. Then, we propose four different advanced methods. Through an extensive experimental evaluation over 7 real-world, established datasets, we show that all the advanced methods outperform to a significant extent both the naïve and the state-of-the-art schema-based ones. We also investigate the relative performance of the advanced methods, providing guidelines on the method selection.

Blocking and Filtering Techniques for Entity Resolution

“…The schema-agnostic progressive methods are suitable for both structured and semi-structured data. They are distinguished into two categories [138]: (i) Inspired by schema-agnostic SN, the sort-based methods sort all entities alphabetically, according to their attribute value tokens. Local Schema-agnostic Progressive SN [138] slides an incremental window over the sorted list of entities.…”

“…They are distinguished into two categories [138]: (i) Inspired by schema-agnostic SN, the sort-based methods sort all entities alphabetically, according to their attribute value tokens. Local Schema-agnostic Progressive SN [138] slides an incremental window over the sorted list of entities. For each window size, it aggregates all distinct comparisons and orders them through a schemaagnostic weighting scheme that considers the number of blocking keys per entity along with the co-occurrence frequency of entity pairs for the current window size.…”

“…All redundant comparisons for each window size are discarded in this way. Global Schema-agnostic Progressive SN [138] does the same, but for a predetermined range of windows, eliminating all redundant comparisons they contain. (ii) Inspired by Meta-blocking, the hash-based methods leverage the blocking graph for the prioritization step.…”

“…(ii) Inspired by Meta-blocking, the hash-based methods leverage the blocking graph for the prioritization step. Progressive Block Scheduling [138] orders the blocks in ascending number of comparisons and then prioritizes all comparisons in the current block, by ordering them in decreasing edge weight, whereas Progressive Profile Scheduling [138] orders entities in decreasing average weight of the corresponding node neighborhood in the blocking graph and then orders all comparisons of the current entity in decreasing edge weight.…”

“…The schema-agnostic methods outperform the schema-based ones with respect to both recall and precision [138]. Yet, they exclusively support static prioritization, defining an immutable processing order that disregards the detection of duplicates.…”