A Scalable Approach to Harvest Modern Weblogs

Abstract

Blogs are one of the most prominent means of communication on the web. Their content, interconnections and influence constitute a unique socio-technical artefact of our times which needs to be preserved. The BlogForever project has established best practices and developed an innovative system to harvest, preserve, manage and reuse blog content. This paper presents the latest developments of the blog crawler which is a key component of the BlogForever platform. More precisely, our work concentrates on techniques to automatically extract content such as articles, authors, dates and comments from blog posts. To achieve this goal, we introduce a simple yet robust and scalable algorithm to generate extraction rules based on string matching using the blog's web feed in conjunction with blog hypertext. Furthermore, we present a system architecture which is characterised by efficiency, modularity, scalability and interoperability with third-party systems. Finally, we conduct thorough evaluations of the performance and accuracy of our system.

Web crawling, analysis and archiving

“…Using the framework we developed in the context of WebGraph-It to enable easy web crawling algorithm implementation (Section 4.3.2), we aim to evolve existing web crawling algorithms and create new. Also, we plan to implement our methods in other open existing web crawlers such as the BlogForever platform [16]. Finally, we aim to launch a public web service via http://webgraph-it.com to provide users with web crawling and webgraph generation services.…”

“…Debian linux operating system [118] for development and production servers, 2. Nginx web server 16 to server static web content, The home page of the ArchiveReady system is presented in Figure 3.5. An overview of the system architecture is presented in Figure 3.4.…”

“…We use the Sorensen-Dice coefficient similarity because it is a string similarity algorithm with the following characteristics: (i) low sensitivity to word ordering, (ii) low sensitivity to length variations, (iii) runs in linear time [16,45].…”

“…Invenio has been originally developed at CERN to run the CERN document server, managing over 1,000,000 bibliographic records in highenergy physics since 2002, covering articles, books, journals, photos, videos, and more. Invenio is being co-developed by an international collaboration comprising institutes such as CERN 14 , DESY 15 , EPFL 16 , FNAL 17 , SLAC 18 and is used by hundreds of institutions world-wide. As it is outside the scope of this thesis, basic Invenio concepts will not be discussed here.…”

“…We aim to evolve the presented Webgraph assisted web crawling methods we presented in Chapter 4 and also create new variants. We plan to implement our methods in other open existing web crawlers such as the BlogForever platform [16]. Finally, we aim to launch a public web service via http://webgraph-it.com to provide users with web crawling and webgraph generation services.…”