A commonly held belief is that traffic engineering and routing changes are infrequent. However, based on our measurements over a number of years of traffic between data centers in one of the largest cloud provider's networks, we found that it is common for flows to change paths at ten-second intervals or even faster. These frequent path and, consequently, latency variations can negatively impact the performance of cloud applications, specifically, latency-sensitive and geo-distributed applications. Our recent measurements and analysis focused on observing path changes and latency variations between different Amazon aws regions. To this end, we devised a path change detector that we validated using both ad hoc experiments and feedback from cloud networking experts. The results provide three main insights: (1) Traffic Engineering (TE) frequently moves (TCP and UDP) flows among network paths of different latency, (2) Flows experience unfair performance, where a subset of flows between two machines can suffer large latency penalties (up to 32% at the 95th percentile) or excessive number of latency changes, and (3) Tenants may have incentives to selfishly move traffic to low latency classes (to boost the performance of their applications). We showcase this third insight with an example using rsync synchronization. To the best of our knowledge, this is the first paper to reveal the high frequency of TE activity within a large cloud provider's network. Based on these observations, we expect our paper to spur discussions and future research on how cloud providers and their tenants can ultimately reconcile their independent and possibly conflicting objectives. Our data is publicly available for reproducibility and further analysis at http://goo.gl/25BKte.

Colocation facilities and Internet eXchange Points (IXPs) provide neutral places for concurrent networks to daily exchange terabytes of data traffic. Although very reliable, these facilities are not immune to failure and may experience difficulties that can have significant impacts on exchanged traffic. In this paper we devise a methodology to identify collocation facilities in traceroute data and to monitor delay and routing patterns between facilities. We also present an anomaly detection technique to report abnormal traffic changes usually due to facilities outages. We evaluate this method with eight months of traceroute data from the RIPE Atlas measurement platform and manually inspect the most prominent events, that are: an IXP outage, a DDoS attack, and a power failure in a facility. These case studies validate the benefits of the proposed system to detect real world outages from traceroute data. We also investigate the impact of anomalies at the metropolitan-level and identify outages that span across up to eight facilities.

Routing hijack attacks have plagued the Internet for decades. After many failed mitigation attempts, recent Internet-wide BGP monitoring infrastructures relying on distributed route collection systems, called route collectors, give us hope that future monitor systems can quickly detect and ultimately mitigate hijacks. In this paper, we investigate the effectiveness of public route collectors with respect to future attackers deliberately engineering longer hijacks to avoid being recorded by route collectors. Our extensive simulations (and attacks we device) show that monitor-based systems may be unable to observe many carefully crafted hijacks diverting traffic from thousands of ASes. Hijackers could predict whether their attacks would propagate to some BGP feeders (i.e., monitors) of public route collectors. Then, manipulate BGP route propagation so that the attack never reaches those monitors. This observation remains true when considering plausible future Internet topologies, with more IXP links and up to 4 times more monitors peering with route collectors. We then evaluate the feasibility of performing hijacks not observed by route collectors in the real-world. We experiment with two classifiers to predict the monitors that are dangerous to report the attack to route collectors, one based on monitor proximities (i.e., shortest path lengths) and another based on Gao-Rexford routing policies. We show that a proximity-based classifier could be sufficient for the hijacker to identify all dangerous monitors for hijacks announced to peer-to-peer neighbors. For hijacks announced to transit networks, a Gao-Rexford classifier reduces wrong inferences by ≥ 91% without introducing new misclassifications for existing dangerous monitors.

The lack of security of the Internet routing protocol (BGP) has allowed attackers to divert Internet traffic and consequently perpetrate service disruptions, monetary frauds, and even citizen surveillance for decades. State-of-the-art defenses rely on geo-distributed BGP monitors to detect rogue BGP announcements. As we show, though, attackers can easily evade detection by engineering their announcements.This paper presents Oscilloscope, an approach to accurately detect BGP hijacks by relying on real-time traffic analysis. As hijacks inevitably change the characteristics of the diverted traffic, the key idea is to track these changes in real time and flag them. The main challenge is that "normal" Internet events (e.g., network reconfigurations, link failures, load balancing) also change the underlying traffic characteristics -and they are way more frequent than hijacks. Naive traffic analyses would hence lead to too many false positives.We observe that hijacks typically target a subset of the prefixes announced by Internet service providers and only divert a subset of their traffic. In contrast, normal events lead to more uniform changes across prefixes and traffic. Oscilloscope uses this observation to filter out non-hijack events by checking whether they affect multiple related prefixes or not.Our experimental evaluation demonstrates that Oscilloscope quickly and accurately detects hijacks in realistic traffic traces containing hundreds of events.