Large-scale Metadata Harvesting—Tools, Techniques and Challenges: A Case Study of National Digital Library (NDL)

Abstract

OAI-PMH enabled open source digital library software like DSpace, EPrints, VuFind, Drupal OAI harvester, PKP harvester had made it possible to harvest massive metadata from different IDR's. IT brought new hope and new opportunities for providing various new services to our library users. This article attempts to explore the tools, techniques and the significant challenges for large-scale metadata harvesting and metadata curation. A recent bibliographic study of Scopus had shown that there is a rapid increase of article publication over the last two decades. "A total of 25,482 publications represent the literary output in different formats, in different subjects, and from various nations" (ul Ajaz Wani & Gul, 2008). All these preprint academic research documents like conference papers, journal article, annual reports, protocols, lecture notes may be already uploaded or needed be upload in various institutional digital repositories (IDR) for long-term digital preservation and reuse. In this study we have harvested the metadata from different such IDRs into a centrally indexed repository for providing a single window search box. Therefore, with this we may dream that day is not far away when we will not need any e-resource subscriptions, as those will be available in our IDR. It will be indeed a great achievement and will be very much helpful to the academic community. However, along with this, a continuous metadata curation is a major intermediate phase, which focuses on the proper mapping of data to metadata. Programmatic curation and manual curations are the two processes done for final curation of the harvested metadata, where both are having their own merits and demerits. This article further focuses on the process workflow of metadata data curation, and the possible challenges need to manage by the librarian for proper indexing of the items.