Web-based Arabic/English duplicate record detection with nested blocking technique

Real-World datasets accumulated over a number of years tend to be incomplete, inconsistent and contain noisy data, this, in turn, will cause an inconsistency of data warehouses. Data owners are having hundred-millions to billions of records written in different languages, hence continuously increases the need for comprehensive, efficient techniques to maintain data consistency and increase its quality. It is known that the data cleaning is a very complex and difficult task, especially for the data written in Arabic as a complex language, where various types of unclean data can occur to the contents. For example, missing values, dummy values, redundant, inconsistent values, misspelling, and noisy data. The ultimate goal of this paper is to improve the data quality by cleaning the contents of Arabic datasets from various types of errors, to produce data for better analysis and highly accurate results. This, in turn, leads to discover correct patterns of knowledge and get an accurate Decision-Making. This approach established based on the merging of different algorithms. It ensures that reliable methods are used for data cleansing. This approach cleans the Arabic datasets based on the multi-level cleaning using Arabic Misspelling Detection, Correction Model (AMDCM), and Decision Tree Induction (DTI). This approach can solve the problems of Arabic language misspelling, cryptic values, dummy values, and unification of naming styles. A sample of data before and after cleaning errors presented. . Currently, he is teaching the master degree students and a supervisor of four master thesis. He is a jury member of a number of PhD and master thesis, as an internal and external examiner in his field of his specialist.

show abstract

“…The system used ten features to classify the character models. Also used [34]. There is a set of cleaning tools available for data cleaning.…”

Section: Related Workmentioning

confidence: 99%

A hybrid Technique for Cleaning Missing and Misspelling Arabic Data in Data Warehouse

Al-Hagery¹,

Alreshoodi²,

Almutairi³

et al. 2019

IJITCS

View full text Add to dashboard Cite

show abstract

“…In this step, datasets are converted to Unicode system to support Arabic language. Several works (Yousef, 2015;Higazy et al, 2013;El-Shishtawy, 2013;Yousef, 2013) used a set of standardization rules for Arabic datasets. These rules consist in replacing a set of characters with their equivalent character.…”

Section: Preprocessingmentioning

confidence: 99%

“…Several ER frameworks have been developed for datasets in Latin and in particular English language such as Febrl (Christen, 2008), TAILOR (Elfeky, Verykios, & Elmagarmid, 2002) and BigMatch (Yancey,2002).Theseframeworksdonotrecognizenon-Latincharactersandinparticular ArabiccharactersbecausetheydonotuseUnicodesystem (Higazy,ElTobely,Yousef,&Sarhan, 2013).Ontheotherhand,developedapproachestosupportERinArabicdatasets (Gueddah,Yousfi, &Belkasmi,2012;Ghafour,El-Bastawissy,&Heggazy,2011;El-Shishtawy,2013;Yousef,2013;Aqeel,Beitzel,Jensen,Grossman,&Frieder,2006)requirematchingrulesortrainingsetdeveloped byanexpert.…”

Section: Introductionmentioning

confidence: 99%

An Unsupervised Entity Resolution Framework for English and Arabic Datasets

Ouhab

Malki²,

Berrabah

et al. 2017

International Journal of Strategic Information Technology and Applications

View full text Add to dashboard Cite

Entity resolution (ER) is an important step in data integration and in many data mining projects; its goal is to identify records that refer to the same real-world entity. Most existing ER frameworks have focused on datasets in Latin-based languages and do not support Arabic language. In this article, the authors present an unsupervised ER framework that supports English and Arabic datasets. Rather than using matching rules developed by an expert or manually labeled training examples, the proposed framework automatically generates its own training set. The generated training set is then used to train a classifier and learn a classification model. Finally, the learned classification model is used to perform ER. The proposed framework was implemented and tested on three Arabic datasets and four English datasets. Experimental results show that the proposed framework is competitive with supervised approaches and outperform recently proposed unsupervised approaches in terms of F-measure.

show abstract

“…Duplicate detection tools such as the Febrl system, TAILOR, and BigMatch were also used in cleaning data. However, Febrl has usability limitations such as slowness, unclear error messages, and complicated installations [17][18][19][20]. The listed programs are rather complex to the average users who do not have experience with programming and language functions.…”

Section: Introductionmentioning

confidence: 99%

Application of Efficient Data Cleaning Using Text Clustering for Semistructured Medical Reports to Large-Scale Stool Examination Reports: Methodology Study (Preprint)

Woo¹,

Kim²,

Cha³

et al. 2018

Preprint

View full text Add to dashboard Cite

BACKGROUND Since medical research based on big data has become more common, the community’s interest and effort to analyze a large amount of semistructured or unstructured text data, such as examination reports, have rapidly increased. However, these large-scale text data are often not readily applicable to analysis owing to typographical errors, inconsistencies, or data entry problems. Therefore, an efficient data cleaning process is required to ensure the veracity of such data. OBJECTIVE In this paper, we proposed an efficient data cleaning process for large-scale medical text data, which employs text clustering methods and value-converting technique, and evaluated its performance with medical examination text data. METHODS The proposed data cleaning process consists of text clustering and value-merging. In the text clustering step, we suggested the use of key collision and nearest neighbor methods in a complementary manner. Words (called values) in the same cluster would be expected as a correct value and its wrong representations. In the value-converting step, wrong values for each identified cluster would be converted into their correct value. We applied these data cleaning process to 574,266 stool examination reports produced for parasite analysis at Samsung Medical Center from 1995 to 2015. The performance of the proposed process was examined and compared with data cleaning processes based on a single clustering method. We used OpenRefine 2.7, an open source application that provides various text clustering methods and an efficient user interface for value-converting with common-value suggestion. RESULTS A total of 1,167,104 words in stool examination reports were surveyed. In the data cleaning process, we discovered 30 correct words and 45 patterns of typographical errors and duplicates. We observed high correction rates for words with typographical errors (98.61%) and typographical error patterns (97.78%). The resulting data accuracy was nearly 100% based on the number of total words. CONCLUSIONS Our data cleaning process based on the combinatorial use of key collision and nearest neighbor methods provides an efficient cleaning of large-scale text data and hence improves data accuracy.

show abstract

Web-based Arabic/English duplicate record detection with nested blocking technique

Cited by 10 publications

References 14 publications

A hybrid Technique for Cleaning Missing and Misspelling Arabic Data in Data Warehouse

A hybrid Technique for Cleaning Missing and Misspelling Arabic Data in Data Warehouse

An Unsupervised Entity Resolution Framework for English and Arabic Datasets

Application of Efficient Data Cleaning Using Text Clustering for Semistructured Medical Reports to Large-Scale Stool Examination Reports: Methodology Study (Preprint)

Contact Info

Product

Resources

About