DDD: Distributed Dataset DNS

Abstract

The advent of inexpensive data storage has resulted in larger and larger datasets as the cost of pruning data becomes more expensive then storing it for future insights. This decreasing cost of storage has also led to the practice of storing data in multiple locations for redundancy. However, without any uniform method of determining link costs to different storage sites, a dataset is not always retrieved from the most cost effective site. Distributed dataset DNS, or DDD, solves this problem in two key ways. The first allows “local” servers to provide meaningful information to a user in order to ensure that they target the location that offers the most advantageous network connection. The second allows other trusted servers to easily gain access to this information in a distributed way. These combined approaches aim to both lower aggregate network bandwidth usage and prevent single points of failure when retrieving dataset pointers.