Turning the Web Into an Effective Knowledge Repository

Abstract

Abstract:To fulfill Vannevar Bush's Memex and Ted Nelson's Hyper-Text vision of a world-size interconnected store of knowledge, there are still quite a few rough-edges to solve. There are no large-scale mechanisms to enforce referential integrity in the WWW. The weight of dynamically generated content w.r.t. static content has progressed enormously. Preserving accessibility to this type of content raises new issues. We propose a system, comprised of a distributed web-proxy and cache architecture, to access and automatically manage web content, static and dynamically generated. It is combined with an implementation of a cyclic distributed garbage collection algorithm for wide-area memory. It correctly handles dynamic content, enforces referential integrity on the web, and is complete w.r.t minimizing storage waste.