Entity matching (EM), which is, the task of identifying records that refer to the same entity, is a critical task when constructing data warehouses. This task is often very expensive at the running time because data must be compared in pairs. This problem becomes more important when dealing with large-scale data. We propose a new parallel algorithm that divides the data using K-Medoid algorithm implemented with Spark framework. The computational experiments are done and show that we can improve the solution of a set of instances in a reduced execution time.

The presence of duplicate records is a major data quality concern in huge datasets. To detect duplicates, entity matching is used as an essential step of the data cleaning process to map records that refer to the same real-world entity. Most of proposed algorithms require labeled data in order to train a classifier. However, we cannot always obtain labeled data. In our paper we propose an unsupervised approach for entity matching problem using an improved version of genetic algorithm. We explain the main improvements added to genetic algorithm and the encoding strategy to encode partitions in the form of a chromosome. Different similarity functions are used to compute similarities between records. The obtained results prove that our proposition stands as a powerful approach in the entity matching field where it outperforms the traditional genetic algorithm based approach.

No abstract

No abstract

Entity Resolution is the task of mapping the records within a database to their corresponding entities. The entity resolution problem presents a lot of challenges because of the absence of complete information in records, variant distribution of records for different entities and sometimes overlaps between records of different entities. In this paper, we have proposed an unsupervised method to solve this problem. The previously mentioned problem is set as a partitioning problem. Thereafter, an optimization algorithm-based technique is proposed to solve the entity resolution problem. The presented approach enables the partitioning of records across entities. A comparative analysis with the genetic algorithm over datasets proves the efficiency of the considered approach.