Profiling Entity Matching Benchmark Tasks

Abstract

Entity matching is a central task in data integration which has been researched for decades. Over this time, a wide range of benchmark tasks for evaluating entity matching methods has been developed. This resource paper systematically complements, profiles, and compares 21 entity matching benchmark tasks. In order to better understand the specific challenges associated with different tasks, we define a set of profiling dimensions which capture central aspects of the matching tasks. Using these dimensions, we create groups of benchmark tasks having similar characteristics. Afterwards, we assess the difficulty of the tasks in each group by computing baseline evaluation results using standard feature engineering together with two common classification methods. In order to enable the exact reproducibility of evaluation results, matching tasks need to contain exactly defined sets of matching and non-matching record pairs, as well as a fixed development and test split. As this is not the case for some widely-used benchmark tasks, we complement these tasks with fixed sets of non-matching pairs, as well as fixed splits, and provide the resulting development and test sets for public download. By profiling and complementing the benchmark tasks, we support researchers to select challenging as well as diverse tasks and to compare matching systems on clearly defined grounds.

Product Matching Lessons and Recommendations from a Real World Application

“…We generate classification models for product matching using three established methodologies: traditional machine leaning models [22], DeepMatcher hybrid models [2], and transformer architecture models [19]. We chose these three approaches for their recent state of the art performance on product matching tasks, and the availability of code from recent publications [2,19,21].…”

“…It is these matching and non-matching pairs together which form the correspondence set we use to train and evaluate our classification systems. We follow the same approach to generating the correspondence sets as used in [21], shown in Algorithm 1. Non-matching pairs are drawn from the Cartesian product of the two datasets being matched, excluding the pairs already labeled as known matches.…”

“…We use two levels of blocking to discard obvious non-matches. The first blocking pass is performed on the primary text attribute(s) using relaxed Jaccard index with inner Levenshtein distance, as performed in [21]. Jaccard index is a set similarity metric, computed for two sets A and B as |A∩B| |A∪B| .…”

“…Primpeli and Bizer [21] proposed five dimensions that can be used to profile matchingrelated challenges. We use these dimensions to form a hypothesis about which matching methods are the most appropriate for our dataset.…”

“…In contrast, our proprietary data has lower schema complexity, lower sparsity, and shorter attribute values. Based upon the groupings presented in [21], we expect our proprietary data to be a good fit for traditional machine learning approaches, while the other datasets may benefit from the deep learning approaches (Note that Primpeli et. al [21] investigated only Random Forest and SVM in their paper).…”