Analysis of Clustering Algorithms to Clean and Normalize Early Modern European Book Titles

Bryer, Evan; Rhujittawiwat, Theppatorn; Comandur, Samyu; Madrid, Vasco; Riley, S. A.; Rose, John R.; Wilder, Colin

doi:10.1145/3451471.3451489

Cited by 2 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The first data set will be one preferable for our new algorithm, and is the same data set we used in our last paper to compare the different clustering algorithms (Bryer et al, 2021). The data set is made up of approximately 1,000 idioms and proverbs taken from English, French, German and Latin.…”

Section: Methodsmentioning

confidence: 99%

“…This dataset has been created and maintained by the Online Computer Library Center (OCLC). A more detailed history of this organization as well as our relationship with them can be found in our previous paper (Bryer et al, 2021). The data contained within the MARC records (in MARC 21 format) stores the metadata of a book, all of the bibliographic and publishing information without containing any of the actual data from the book.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Spelling Based Ranked Clustering Algorithm to Clean and Normalize Early Modern European Book Titles

2021

Proceedings of the 15th International Conference on Computer Graphics, Visualization, Computer Vision and Image Processing (CGV

View full text Add to dashboard Cite

The goal of this paper is to modify an existing clustering algorithm with the use of the Hunspell spell checker to specialize it for the use of cleaning early modern European book title data. Duplicate and corrupted data is a constant concern for data analysis, and clustering has been identified to be a robust tool for normalizing and cleaning data such as ours. In particular, our data comprises over 5 million books published in European languages between 1500 and 1800 in the Machine-Readable Cataloging (MARC) data format from 17,983 libraries in 123 countries. However, as each library individually catalogued their records, many duplicative and inaccurate records exist in the data set. Additionally, each language evolved over the 300 year period we are studying, and as such many of the words had their spellings altered. Without cleaning and normalizing this data, it would be difficult to find coherent trends, as much of the data may be missed in the query. In previous research, we have identified the use of Prediction by Partial Matching to provide the most increase in base accuracy when applied to dirty data of similar construct to our data set. However, there are many cases in which the correct book title may not be the most common, either when only two values exist in a cluster or the dirty title exists in more records. In these cases, a language agnostic clustering algorithm would normalize the incorrect title, and lower the overall accuracy of the data set. By implementing the Hunspell spell checker into the clustering algorithm, using it to rank clusters by the number of words not found in their dictionary, we can drastically lower the cases of this occurring. Indeed, this ranking algorithm proved to increase the overall accuracy of the clustered data by as much as 25% over the unmodified Prediction by Partial Matching algorithm.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Spelling Based Ranked Clustering Algorithm to Clean and Normalize Early Modern European Book Titles

2021

Proceedings of the 15th International Conference on Computer Graphics, Visualization, Computer Vision and Image Processing (CGV

View full text Add to dashboard Cite

show abstract

“…Evan Bryer wraz z zespołem (Bryer et al, 2021) opracowali metodę deduplikacji rekordów metadanych w oparciu o strategię oczyszczania i analizę skupień z wykorzystaniem metod i technologii uczenia maszynowego. Przedmiotem badań była baza ponad 5 mln rekordów w formacie MARC 21 dla wydawnictw zwartych opublikowanych między XVI a XIX w. i dostępnych za pośrednictwem bazy WorldCat.…”

Section: Wymiar Ontologiczny Bdsunclassified

Bibliographic Data Science – Konceptualizacja Obszaru Badawczego

Roszkowski

2022

View full text Add to dashboard Cite

Teza/cel artykułu – Przedmiotem artykułu jest nowy obszar badaw- czy o nazwie bibliographic data science, który charakteryzuje się zastosowaniem metod i technologii data science w badaniach nad zasobami katalogów bibliotecz- nych i bibliografii. Celem artykułu jest próba konceptualizacji bibliographic data science. Metody – W warstwie metodologicznej przeprowadzone badania opierają się na analizie i krytyce piśmiennictwa dokumentującego badania prowadzone pod szyldem bibliographic data science oraz koncepcji analizy domen. Wyniki – U podstaw bibliographic data science leży pragmatyczna postawa wobec badań zasobów bibliograficznych z wykorzystaniem metod data science realizowanych w humanistyce cyfrowej. W ramach tego obszaru badawczego stawia się pro- blemy właściwe dla dyscyplin tworzących cyfrową humanistykę oraz podejmuje zagadnienia metodologiczne związane z optymalizacją jakości danych bibliogra- ficznych oraz ich harmonizacją. Silne związki bibliographic data science z cyfrową humanistyką widoczne są również w społeczności skupionej wokół tego obszaru badawczego.

show abstract

Analysis of Clustering Algorithms to Clean and Normalize Early Modern European Book Titles

Cited by 2 publications

References 9 publications

Spelling Based Ranked Clustering Algorithm to Clean and Normalize Early Modern European Book Titles

Spelling Based Ranked Clustering Algorithm to Clean and Normalize Early Modern European Book Titles

Bibliographic Data Science – Konceptualizacja Obszaru Badawczego

Contact Info

Product

Resources

About