I/O-efficient techniques for computing pagerank

Abstract

Over the last few years, most major search engines have integrated link-based ranking techniques in order to provide more accurate search results. One widely known approach is the Pagerank technique, which forms the basis of the Google ranking scheme, and which assigns a global importance measure to each page based on the importance of other pages pointing to it. The main advantage of the Pagerank measure is that it is independent of the query posed by a user; this means that it can be precomputed and then used to optimize the layout of the inverted index structure accordingly. However, computing the Pagerank measure requires implementing an iterative process on a massive graph corresponding to billions of web pages and hyperlinks.In this paper, we study I/O-efficient techniques to perform this iterative computation. We derive two algorithms for Pagerank based on techniques proposed for out-of-core graph algorithms, and compare them to two existing algorithms proposed by Haveliwala. We also consider the implementation of a recently proposed topic-sensitive version of Pagerank. Our experimental results show that for very large data sets, significant improvements over previous results can be achieved on machines with moderate amounts of memory. On the other hand, at most minor improvements are

Efficient Parallel Computation of PageRank

“…Because pages only link to a few others (the link matrix is sparse), this results in much lower memory requirements of the link structure, in the magnitude of | L | · n −1 (n = average outdegree). Of course, compression techniques [14] or disk-based "swapping" [9,5] can improve the space requirements even further. But with the permanent growth of the web, even such techniques will soon hit memory limits of a single computer, or unacceptably slow down the computation process.…”

“…We can therefore also write Equation 1 in matrix terms as follows: Equation 1 also represents the linear system representation of this computation using the Jacobi method. This enables the consideration of using other stationary iterative solvers, such as the Gauß-Seidel method, which was said not to be efficiently parallelizable here [1,5]. Actually, there already are parallel Gauss-Seidel implementations for certain scenarios such as the one described in [13], using block-diagonally-bordered matrices; however, they all admit their approach was designed for a static matrix; after each modification, a specific preprocessing (sorting) step is required, which can take longer than the real computation.…”

“…Although improvements for a centralized computation of PageRank have been researched in detail [1,9,11,5,12,20,15], approaches on distributing it over several computers have caught researchers' attention only recently. In this paper we introduce a new approach to computing the exact PageRank in a parallel fashion.…”

Agnostic topology-based spam avoidance in large-scale web crawls

“…Finally, TSE reads D = 2F bytes and maintains Ω = pF , without ever requiring that all supporter counts fit in RAM. Note that since both BV and QVS are iterative methods similar to PageRank, their Ω can be reduced at the expense of certain offline pre-processing on the graph [8]. We do not consider this in our comparison.…”

An Efficient Algorithm and Its Parallelization for Computing PageRank

“…However, as pointed out by [6], even the best compression scheme requires about .6 bytes per hyperlink, which still results in an exceedingly large space requirement. Others are to design I/O-efficient algorithms, such as [9,10], which can handle any size of data without any particular memory requirement. Also aiming at reducing the cost of each iteration, many other works, such as [11,12], combine some linear algebra techniques to reduce the computational cost of each iteration.…”