Mapping the Blogosphere with RSS-Feeds

Abstract

The massive adoption of social media has provided new ways for individuals to express their opinions online. The blogosphere, an inherent part of this trend, contains a vast array of information about a variety of topics. It is thus a huge think tank that creates an enormous and ever-changing archive of open source intelligence. Modeling and mining this vast pool of data to extract, exploit and describe meaningful knowledge in order to leverage (content-related) structures and dynamics of emerging networks within the blogosphere is the higher-level aim of the research presented here. This paper focuses on this project's initial phase, in which the abovementioned data of interest needs to be collected and made available offline for further analyses. Our proprietary development of a tailor-made feed-crawler meets exactly this need. The main concept, the techniques and the implementation details of the crawler thus form the main interest of this paper and furthermore provide the basis for future project phases.

Where do bloggers blog? Platform transitions within the historical Dutch blogosphere

“…Besides the notion of the blogosphere as a space for discourse, other definitions stress the formalistic characteristics of the blogosphere as an interlinked set of blogs which "allows for the networked, decentralised, distributed discussion and deliberation on a wide range of topics" (Bruns, et al, 2009). A complimentary approach to the blogosphere as an interlinked set of blogs looks at how blogs are "embedded into a much bigger picture: a segmented and independent public that dynamically evolves and functions according to its own rules and with ever-changing protagonists, a network also known as the 'blogosphere'" (Bross, et al, 2010). Following this line of thinking, i.e., blogs which are embedded in a larger networked ecology with shifting protagonists, the blogosphere may also be defined by including the actors they link to in their networked ecology: "The notion of a mini-blogsphere additionally rests on the extent to which the set of blogs doing an issue are interconnected by links and/or by textual referencing.…”

“…In other words, to become visible, the image of the blogosphere must be constructed, either by blogosphere related services such as directories, Web rings and blog search or by academic network visualizations. By means of similar techniques such as contemporary blog related services, network visualizations may be constructed by employing RSS feed-crawlers to fetch the contentcurrent and newly updated blog posts and their links -of blogs using their feeds (Bross, et al, 2010) or by using Web crawlers for network analysis. Crawling and network analysis may be used for widely varying analytical purposes, such as the IssueCrawler for issue network analysis to track conversation patterns in the blogosphere (Bruns, 2007); crawling the front page of blogs to reflect blogroll communities (Adamic and Glance, 2005); and, large scale grouping of linked blogs to define clusters of shared informational worlds (Kelly and Etling, 2008).…”

RSS-Crawler Enhancement for Blogosphere-Mapping

“…We perceive it as nearsighted to base research like the ones mentioned before on data of external services like Technorati, BlogPulse or Spinn3r [6]. We also have ambitious plans of how to ultimately use blog data [3] -we at least make the effort of setting up our own crawling framework to ensure and prove that the data employed in our research has the quantity, structure, format and quality required and necessary [4].…”

“…To allow the processing of the enormous amount of content in the blogosphere, it was necessary to make that content available offline for further analysis. The first prototype of our feedcrawler completed this assignment along the milestones specified in the initial project phase [4]. However, it soon became apparent that a considerable amount of optimization would be necessary to fully account for the strong distinction between crawling regular web pages and mining the highly dynamic environment of the blogosphere.…”

“…This interconnected think tank thus creates an enormous and ever-changing archive of open source intelligence [12]. Modeling and mining the vast pool of data generated by the blogosphere to extract, exploit and represent meaningful knowledge in order to leverage (content-related) structures of emerging social networks residing in the blogosphere were the main objective of the projects initial phase [4].…”