The Link Database: fast access to graphs of the Web

Randall, Karen; Stata, Raymie; Wickremesinghe, Rajiv; Wiener, Joshua

doi:10.1109/dcc.2002.999950

Cited by 66 publications

(52 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can be easily seen that including dangling nodes does not add any storage overhead since a dangling node does not appear in the link file as a source id. Thus, our approach does not change the I/O cost model in (6).…”

Section: Dangling Nodes Treatmentmentioning

confidence: 99%

“…Because the major obstacle of speeding up each iteration lies in the size of data, it is natural to compress the data such that it fits into the main memory, as by [6,7,8]. However, as pointed out by [6], even the best compression scheme requires about .6 bytes per hyperlink, which still results in an exceedingly large space requirement. Others are to design I/O-efficient algorithms, such as [9,10], which can handle any size of data without any particular memory requirement.…”

Section: Related Workmentioning

confidence: 99%

“…Updating the source vector at the end adds another pass of v. Therefore, the cost (referred to as C block ) is (6) where is a small positive number due to the partition overhead.…”

Section: The Blocking Algorithmmentioning

confidence: 99%

See 2 more Smart Citations

An Efficient Algorithm and Its Parallelization for Computing PageRank

Qiao

Jones

Thrall

2007

Computational Science – ICCS 2007

View full text Add to dashboard Cite

Abstract. In this paper, an efficient algorithm and its parallelization to compute PageRank are proposed. There are existing algorithms to perform such tasks. However, some algorithms exclude dangling nodes which are an important part and carry important information of the web graph. In this work, we consider dangling nodes as regular web pages without changing the web graph structure and therefore fully preserve the information carried by them. This differs from some other algorithms which include dangling nodes but treat them differently from regular pages for the purpose of efficiency. We then give an efficient algorithm with negligible overhead associated with dangling node treatment. Moreover, the treatment poses little difficulty in the parallelization of the algorithm.

show abstract

Section: Dangling Nodes Treatmentmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

An Efficient Algorithm and Its Parallelization for Computing PageRank

Qiao

Jones

Thrall

2007

Computational Science – ICCS 2007

View full text Add to dashboard Cite

show abstract

“…The WebGraph compression method is indeed the most successful member of a family of approaches to compress Web graphs based on their statistical properties [5,7,1,23,21,20]. It allows fast extraction of the neighbors of a page while spending just a few bits per link (about 2 to 6, depending on the desired navigation performance).…”

Section: Introductionmentioning

confidence: 99%

k2-Trees for Compact Web Graph Representation

Brisaboa

Ladra

Navarro

2009

Lecture Notes in Computer Science

106

View full text Add to dashboard Cite

Abstract. This paper presents a Web graph representation based on a compact tree structure that takes advantage of large empty areas of the adjacency matrix of the graph. Our results show that our method is competitive with the best alternatives in the literature, offering a very good compression ratio (3.3-5.3 bits per link) while permitting fast navigation on the graph to obtain direct as well as reverse neighbors (2-15 microseconds per neighbor delivered). Moreover, it allows for extended functionality not usually considered in compressed graph representations.

show abstract

“…Since disk access is (by five orders of magnitude) slower than main memory access, this leads to unacceptable retrieval times. To mitigate this problem, several compression techniques have been proposed for large graphs, aimed at reducing the number of bits per link required in graph representation [1][2][3][4][5][6][7].…”

Section: Introductionmentioning

confidence: 99%

Graph Compression by BFS

2009

View full text Add to dashboard Cite

Abstract:The Web Graph is a large-scale graph that does not fit in main memory, so that lossless compression methods have been proposed for it. This paper introduces a compression scheme that combines efficient storage with fast retrieval for the information in a node. The scheme exploits the properties of the Web Graph without assuming an ordering of the URLs, so that it may be applied to more general graphs. Tests on some datasets of use achieve space savings of about 10% over existing methods.

show abstract

The Link Database: fast access to graphs of the Web

Cited by 66 publications

References 10 publications

An Efficient Algorithm and Its Parallelization for Computing PageRank

An Efficient Algorithm and Its Parallelization for Computing PageRank

k2-Trees for Compact Web Graph Representation

Graph Compression by BFS

Contact Info

Product

Resources

About