Measuring Diversity of a Domain-Specific Crawl

Abstract

In this work we present various metrics to measure diversity of a domain-specic crawl. We evaluate these metrics using domainspecic crawl originated from ODP URLs and nd that these metrics are indeed able to capture diversity. We argue that these metrics can be used for comparing seed sets and crawling strategies with respect to diversity.