Scaling a monitoring infrastructure for the Akamai network

Abstract

We describe the design of, and experience with, Query, a monitoring system that supports the Akamai EdgePlatform. Query is a foundation of Akamai's approach to administering its distributed computing platform, allowing administrators, operations staff, developers, customers, and automated systems near real-time access to data about activity in Akamai's network. Users extract information regarding the current state of the network via a SQL-like interface. Versions of Query have been deployed since the inception of Akamai's platform, and it has scaled to support a distributed platform of 60,000+ servers, collecting over 200 gigabytes of data and answering over 30,000 queries approximately every 2 minutes.

Accounting and Accountability in Content Distribution Architectures

“…However, the ability to monitor all the logging information generated across distributed trusted and untrusted infrastructure is a major challenge [Ko et al 2011;Repantis et al 2010]. Consequently, a major obstacle is to identify what accounting and accountability information should be gathered based on the content distribution infrastructure (trusted or untrusted).…”

“…We give a brief overview of unstructured and structured P2P architectures such as Napster, BitTorrent, and the Kad network (based on Kademlia DHT) and their methodology for collecting accounting information. However, the main drawbacks of using P2P infrastructures for content distribution are the possibility of pollution attacks, content corruption, and lack of trustable accounting information [Kamvar et al 2003;Dhungel et al 2007;Rodriguez et al 2006;. Consequently, we also identify how accountability information can be gathered and utilized by P2P systems based on the PeerReview [Haeberlen et al 2007] system.…”

“…Additionally, content providers want to ensure CDN systems are meeting service-level agreements (SLAs), as poor end-user QoE can have repercussions such as lower revenues and lower user subscriptions. CDN systems also depend on logging information to help troubleshoot and debug system and network faults, resolve security vulnerabilities, and bill customers [Repantis et al 2010].…”

“…Nygren et al [2010]. However, the amount of logging information gathered can consume massive resources such as in Akamai's Query system [Repantis et al 2010]. For example, in 2010, Akamai processed over 100TB of logs per day collected from edge servers [Nygren et al 2010].…”

“…For example, Akamai allows content providers to view information such as user engagement, user QoE, advertising, user demographics, and network performance metrics [Nygren et al 2010]. Furthermore, the Akamai Query system gathers extensive information about distributed surrogate servers to detect system and network anomalies, monitor system usage, raise alarms, and troubleshoot problems, and it is available to Akamai staff and indirectly to Akamai customers [Repantis et al 2010]. In contrast, YouTube only provides accounting information on advertising, user engagement, and user demographics, while CoralCDN provides no accounting information at all to content providers.…”