Cross Language Duplicate Record Detection in Big Data

Yousef, Ahmed H.

doi:10.1007/978-3-319-11056-1_5

Cited by 6 publications

(5 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this step, datasets are converted to Unicode system to support Arabic language. Several works (Yousef, 2015;Higazy et al, 2013;El-Shishtawy, 2013;Yousef, 2013) used a set of standardization rules for Arabic datasets. These rules consist in replacing a set of characters with their equivalent character.…”

Section: Preprocessingmentioning

confidence: 99%

See 1 more Smart Citation

An Unsupervised Entity Resolution Framework for English and Arabic Datasets

Ouhab

Malki²,

Berrabah

et al. 2017

International Journal of Strategic Information Technology and Applications

View full text Add to dashboard Cite

Entity resolution (ER) is an important step in data integration and in many data mining projects; its goal is to identify records that refer to the same real-world entity. Most existing ER frameworks have focused on datasets in Latin-based languages and do not support Arabic language. In this article, the authors present an unsupervised ER framework that supports English and Arabic datasets. Rather than using matching rules developed by an expert or manually labeled training examples, the proposed framework automatically generates its own training set. The generated training set is then used to train a classifier and learn a classification model. Finally, the learned classification model is used to perform ER. The proposed framework was implemented and tested on three Arabic datasets and four English datasets. Experimental results show that the proposed framework is competitive with supervised approaches and outperform recently proposed unsupervised approaches in terms of F-measure.

show abstract

Section: Preprocessingmentioning

confidence: 99%

“…He served as full professor and the head of Evolutionary Engineering and Distributed Information Systems Laboratory (EEDIS Lab.) at Djillali Liabes University of Sidi Bel-Abbes, Algeria (2002-2015…”

mentioning

confidence: 99%

An Unsupervised Entity Resolution Framework for English and Arabic Datasets

Ouhab

Malki²,

Berrabah

et al. 2017

International Journal of Strategic Information Technology and Applications

View full text Add to dashboard Cite

show abstract

“…In this experiment, a comparison between the features allowed in the proposed web-based DRD framework and FEBRL is illustrated. This comparison appears in [60] as a part from their study for the available frameworks that perform DRD. The major advantages of the proposed DRD framework over FEBRL are: the availability of enhancing system behavior through real experiments by the users, allowing bi-lingual processing and introducing the sequential blocking technique instead of the current available indexing techniques.…”

Section: Experiments 3: Proposed Framework Features Compared To Febrlmentioning

confidence: 99%

“…Many researches in Record Linkage/Duplicate Detection have been developed and introduced. Some of them were about providing a complete framework or implementing techniques/algorithms that handle a specific stage in DRD [60]. The general steps for record linkage/duplicate detection [17] are; first is data cleaning and standardization where input data is converted into a well-defined form.…”

Section: Introductionmentioning

confidence: 99%

Web-Based Duplicate Records Detection With Arabic Language Enhancement

Higazy

Sarhan

Eltobely

2015

Journal of Engineering Research

View full text Add to dashboard Cite

“…There are four dimensions of data quality standards: availability, usability, reliability, and relevance [3][4][5] . Reliability is characterized as the trustworthiness of data, including accuracy, consistency, completeness, and integrity 11 . The reliability of BD is difficult to achieve due to its characteristics and complex technology and architecture for processing data [3][4][5] .…”

Section: Introductionmentioning

confidence: 99%

Testing Framework for Big Data: A Case Study of Telecom Sector of Pakistan

Nasir

Imtiaz

et al. 2022

Preprint

View full text Add to dashboard Cite

Big Data has emerged as an asset for business practices and marketing strategies due to its data analysis capabilities. For an effective decision-making process, reliable data is critically important. Unreliable data results in wrong decision making ultimately leading to a negative reputation, customer churn, and financial loss. Therefore, testing Big Data is vital to ensure the reliability of data. Defining test strategies and setting up a testing environment are complex and challenging tasks. In this research, we tend to focus on data reliability, since a wrong decision based on unreliable data can have huge negative consequences. The testing framework for Big Data is validated via an exploratory case study in the telecommunication sector. Three-layer architecture for Big Data testing is used, namely Ingestion, Preparation, and Access. Errors are injected to verify fault injection in the ingestion and processing of data files of the Hadoop framework and technology stack. The results of the testing framework show errors in three layers of Big Data. The testing framework effectively detects data faults in Big Data,and a total of 76.92% errors in the presentation layer of total failed test cases are identified. 48.72% of test cases are developed to detect errors to check data accuracy, and after execution of these test cases 31.58% errors are detected. The highest proportion of errors detected are related to the completeness, which is 66.67% of the number of errors injected for completeness. The results show the effectiveness of the framework to identify accuracy and completeness errors.

show abstract

Cross Language Duplicate Record Detection in Big Data

Cited by 6 publications

References 27 publications

An Unsupervised Entity Resolution Framework for English and Arabic Datasets

An Unsupervised Entity Resolution Framework for English and Arabic Datasets

Web-Based Duplicate Records Detection With Arabic Language Enhancement

Testing Framework for Big Data: A Case Study of Telecom Sector of Pakistan

Contact Info

Product

Resources

About