Network Monitoring as a Streaming Analytics Problem

Gupta, Arpit; Birkner, Rüdiger; Canini, Marco; Feamster, Nick; Mac-Stoker, Chris; Willinger, Walter

doi:10.1145/3005745.3005748

Cited by 41 publications

(18 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior switch-based network measurement systems [18,19,32,23,40,22] are constrained by their raw data, sourced from limited measurement support in existing switches, e.g., NetFlow, match-action rules, and packet mirroring. More recently, Gupta et al [22] propose to partition monitoring queries between switches and a stream processor (e.g., Spark streaming [12]), iteratively refining the set of packets captured through match-action rules in the switch.…”

Section: Related Workmentioning

confidence: 99%

“…More recently, Gupta et al [22] propose to partition monitoring queries between switches and a stream processor (e.g., Spark streaming [12]), iteratively refining the set of packets captured through match-action rules in the switch. In contrast, by designing new switch primitives, we enable operators to answer a more diverse set of performance questions.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Hardware-Software Co-Design for Network Performance Measurement

Narayana

Sivaraman

Nathan

et al. 2016

Proceedings of the 15th ACM Workshop on Hot Topics in Networks

View full text Add to dashboard Cite

Diagnosing performance problems in networks is important, for example to determine where packets experience high latency. However, existing diagnostic tools are constrained by limited switch mechanisms for measurement. As a result, operators use endpoint information to indirectly infer root causes for performance issues.Instead of designing piecemeal solutions to work around limited switch mechanisms, we believe that the right approach is to co-design language abstractions and switch hardware primitives for performance measurement. This approach provides confidence that the switch primitives are useful for a variety of existing and unanticipated use cases.We present a declarative query language that allows operators to ask a diverse set of network performance questions. We show that these queries can be implemented efficiently in switch hardware using a programmable key-value store primitive. Our preliminary evaluations show that our hardware design incurs modest additional chip area relative to existing switching chips, suggesting that it is a practical solution for network performance measurement.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Hardware-Software Co-Design for Network Performance Measurement

Narayana

Sivaraman

Nathan

et al. 2016

Proceedings of the 15th ACM Workshop on Hot Topics in Networks

View full text Add to dashboard Cite

show abstract

“…Information in packet headers naturally constitute key-value tuples (e.g., source and destination IP address, and other header values). This structure lends itself to a tuple-based abstraction [16]. Of course, an operator may want to write queries based on information that is not in the IP packet header, such as the application protocol, or DNS query type.…”

Section: Dataflow Queries On Tuplesmentioning

confidence: 99%

“…Trumpet [31] and Pathdump [47] offload query processing to end-hosts (VMs in data center networks) but not to switches. Our previous work [16] proposed a telemetry system that partitions a single query across a stream processor and switch, but it only considers switches with fixed-function chipsets, and requires the network operators to specify the refinement and partitioning plans manually. In contrast, Sonata supports programmable switches and employs a sophisticated query planner to automatically partition and refine multiple queries.…”

Section: Related Workmentioning

confidence: 99%

Sonata

Gupta

Harrison

Canini

et al. 2018

Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

Self Cite

206

View full text Add to dashboard Cite

Managing and securing networks requires collecting and analyzing network traffic data in real time. Existing telemetry systems do not allow operators to express the range of queries needed to perform management or scale to large traffic volumes and rates. We present Sonata, an expressive and scalable telemetry system that coordinates joint collection and analysis of network traffic. Sonata provides a declarative interface to express queries for a wide range of common telemetry tasks; to enable real-time execution, Sonata partitions each query across the stream processor and the data plane, running as much of the query as it can on the network switch, at line rate. To optimize the use of limited switch memory, Sonata dynamically refines each query to ensure that available resources focus only on traffic that satisfies the query. Our evaluation shows that Sonata can support a wide range of telemetry tasks while reducing the workload for the stream processor by as much as seven orders of magnitude compared to existing telemetry systems.

show abstract

“…The rapid growth of technology increases application areas day by day. In recent years, the developed application areas such as social networks [1], electronic business [2], cloud computing [3,4], computer networks measurement [5][6][7], internet of things applications [8], etc. are generating large volume data [9].…”

Section: Introductionmentioning

confidence: 99%

An Approach for Streaming Data Feature Extraction Based on Discrete Cosine Transform and Particle Swarm Optimization

Aydoğdu

Ekinci

2020

Symmetry

View full text Add to dashboard Cite

Incremental feature extraction algorithms are designed to analyze large-scale data streams. Many of them suffer from high computational cost, time complexity, and data dependency, which adversely affects the processing of the data stream. With this motivation, this paper presents a novel incremental feature extraction approach based on the Discrete Cosine Transform (DCT) for the data stream. The proposed approach is separated into initial and sequential phases, and each phase uses a fixed-size windowing technique for processing the current samples. The initial phase is performed only on the first window to construct the initial model as a baseline. In this phase, normalization and DCT are applied to each sample in the window. Subsequently, the efficient feature subset is determined by a particle swarm optimization-based method. With the construction of the initial model, the sequential phase begins. The normalization and DCT processes are likewise applied to each sample. Afterward, the feature subset is selected according to the initial model. Finally, the k-nearest neighbor classifier is employed for classification. The approach is tested on the well-known streaming data sets and compared with state-of-the-art incremental feature extraction algorithms. The experimental studies demonstrate the proposed approach’s success in terms of recognition accuracy and learning time.

show abstract

Network Monitoring as a Streaming Analytics Problem

Cited by 41 publications

References 21 publications

Hardware-Software Co-Design for Network Performance Measurement

Hardware-Software Co-Design for Network Performance Measurement

Sonata

An Approach for Streaming Data Feature Extraction Based on Discrete Cosine Transform and Particle Swarm Optimization

Contact Info

Product

Resources

About