Duplicate Detection Using Unsupervised Random Forests: A Preliminary Analysis

Omar, Zaturrawiah Ali; Bakar, Mohd Aftar Abu; Zamzuri, Zamira Hasanah; Ariff, Noratiqah Mohd

doi:10.1109/aidas56890.2022.9918724

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We started off with these two datasets in this study since they contain fields of mixed data types (numbers and strings). Based on the initial study that we performed in [32], any number values were set as integer data types and strings as characters. Both datasets have a one-to-one relationship, which means that duplicate records have only one match.…”

Section: Datasetmentioning

confidence: 99%

Training Data Selection for Record Linkage Classification

et al. 2023

Self Cite

View full text Add to dashboard Cite

This paper presents a new two-step approach for record linkage, focusing on the creation of high-quality training data in the first step. The approach employs the unsupervised random forest model as a similarity measure to produce a similarity score vector for record matching. Three constructions were proposed to select non-match pairs for the training data, with both balanced (symmetry) and imbalanced (asymmetry) distributions tested. The top and imbalanced construction was found to be the most effective in producing training data with 100% correct labels. Random forest and support vector machine classification algorithms were compared, and random forest with the top and imbalanced construction produced an F1-score comparable to probabilistic record linkage using the expectation maximisation algorithm and EpiLink. On average, the proposed approach using random forests and the top and imbalanced construction improved the F1-score by 1% and recall by 6.45% compared to existing record linkage methods. By emphasising the creation of high-quality training data, this new approach has the potential to improve the accuracy and efficiency of record linkage for a wide range of applications.

show abstract

Section: Datasetmentioning

confidence: 99%

Training Data Selection for Record Linkage Classification

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…The field of duplication and its treatment methods is immense in the literature. It is addressed from different perspectives and domains, started from an overview of the duplication and techniques [21]- [23], improving the detection techniques [24], [25], evaluating the impact of the duplicates [26], and proposing new frameworks [27], [28] and methods [29], [30] to effectively enhance the detection process. However, little research addresses the impact of duplicates on the analysis results.…”

Section: Related Workmentioning

confidence: 99%

Tracking The Sensitivity of The Learning Models Toward Exact and Near Duplicates

Gabr¹,

Helmy²,

Elzanfaly³

2022

IJACSA

View full text Add to dashboard Cite

Most real-world datasets contaminated by quality issues have a severe effect on the analysis results. Duplication is one of the main quality issues that hinder these results. Different studies have tackled the duplication issue from different perspectives. However, revealing the sensitivity of supervised and unsupervised learning models under the existence of different types of duplicates, deterministic and probabilistic, is not broadly addressed. Furthermore, a simple metric is used to estimate the ratio of both types of duplicates regardless of the probability by which the record is considered duplicate. In this paper, the sensitivity of five classifiers and four clustering algorithms toward deterministic and probabilistic duplicates with different ratios (0% -15%) is tracked. Five evaluation metrics are used to accurately track the changes in the sensitivity of each learning model, MCC, F1-Score, Accuracy, Average Silhouette Coefficient, and DUNN Index. Also, a metric to measure the ratio of probabilistic duplicates within a dataset is introduced. The results revealed the effectiveness of the proposed metric that reflects the ratio of probabilistic duplicates within the dataset. All learning models, classification, and clustering models are differently sensitive to the existence of duplicates. RF and Kmeans are positively affected by the existence of duplicates which means that their performce increase as the percentage of duplicates increases. Furthermore, the rest of classifiers and clustering algorithms are sensitive toward duplicates existence, especially within high percentage that negatively affect their performance.

show abstract

Duplicate Detection Using Unsupervised Random Forests: A Preliminary Analysis

Cited by 2 publications

References 0 publications

Training Data Selection for Record Linkage Classification

Training Data Selection for Record Linkage Classification

Tracking The Sensitivity of The Learning Models Toward Exact and Near Duplicates

Contact Info

Product

Resources

About