RDMA-Based Apache Storm for High-Performance Stream Data Processing

Zhang, Ziyu; Liu, Zitan; Jiang, Qingcai; Chen, Junshi; An, Hong

doi:10.1007/s10766-021-00696-0

Cited by 9 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to the IBTA specifications [8], the QP transmission types of RDMA can be divided into four categories based on two dimensions: whether they are reliable and whether they establish a connection. These four categories are RD, RC, Unreliable Datagram (UD), and Unreliable Connection (UC) [31]. Currently, there are mature implementation solutions for RC, UD, and UC, while the research on RD type is only at the definition level [2].…”

Section: Rdma Basicmentioning

confidence: 99%

An Optimized RDMA QP Communication Mechanism for Hyperscale AI Infrastructure

Wang,

Lin,

Zhang

et al. 2024

Preprint

View full text Add to dashboard Cite

The current artificial intelligence (AI) infrastructure widely employs remote direct memory access (RDMA) protocol for high-performance communication in networks, utilizing Reliable Connection (RC)-based Queue Pairs (QP) to ensure end-to-end correct and ordered data transmission. However, as the scale of AI infrastructure continues to expand, this RC-based QP communication mechanism faces deficiencies in scalability and is prone to congestion, resulting in degraded network transfer performance. In this paper, we propose an optimized RDMA QP communication mechanism to address scalability and congestion issues in hyper-scale AI infrastructure networks. Firstly, we replace RC-based QPs with Reliable Datagram (RD)-based QPs and propose a new reliable mechanism to address scalability problems, eliminating the need for repetitive QP establishment by AI processes during external communication. Additionally, to mitigate congestion caused by a single path, we implement multipath data transmission by introducing a new unordered reception method in the network software stack. Through experiments and simulation tests, the optimized RDMA QP communication in large-scale AI infrastructure exhibits exceptional scalability and significantly reduces the occurrence of congestion, resulting in an overall network performance improvement of over 15%.

show abstract

Section: Rdma Basicmentioning

confidence: 99%

An Optimized RDMA QP Communication Mechanism for Hyperscale AI Infrastructure

Wang,

Lin,

Zhang

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Figure 6 shows different irregular traffic pattern detection. The architecture in [98] allows for network operators to estimate the flow size of encrypted data at multi-Gbps line rates using samples and sketching mechanisms. It also helps in understanding the behavior of VPN-buffered traffic.…”

Section: [92]mentioning

confidence: 99%

“…Evaluation studies depend on the track time and starting point, achieving more than a 90% precision for the content classification of a given service provider in the best case. The examiner The architecture in [98] allows for network operators to estimate the flow size of encrypted data at multi-Gbps line rates using samples and sketching mechanisms. It also helps in understanding the behavior of VPN-buffered traffic.…”

Section: [92]mentioning

confidence: 99%

“…However, new packets arrive 10 times faster than main memory access times while utilizing only one processor core [ 91 ]. Apache storm used the Netty communication component [ 98 ], a TCP/IP protocol stack applied for an asynchronous server, and a client framework that decreased efficiency due to context switching and memory copying. It increased the IP over the InfiniBand communication mode on the CPU load.…”

Section: Ddos Attack Detection In High-speed Networkmentioning

confidence: 99%

See 1 more Smart Citation

High-Speed Network DDoS Attack Detection: A Survey

Haseeb-ur-rehman,

Aman,

Hasan

et al. 2023

Sensors

View full text Add to dashboard Cite

Having a large number of device connections provides attackers with multiple ways to attack a network. This situation can lead to distributed denial-of-service (DDoS) attacks, which can cause fiscal harm and corrupt data. Thus, irregularity detection in traffic data is crucial in detecting malicious behavior in a network, which is essential for network security and the integrity of modern Cyber–Physical Systems (CPS). Nevertheless, studies have shown that current techniques are ineffective at detecting DDoS attacks on networks, especially in the case of high-speed networks (HSN), as detecting attacks on the latter is very complex due to their fast packet processing. This review aims to study and compare different approaches to detecting DDoS attacks, using machine learning (ML) techniques such as k-means, K-Nearest Neighbors (KNN), and Naive Bayes (NB) used in intrusion detection systems (IDSs) and flow-based IDSs, and expresses data paths for packet filtering for HSN performance. This review highlights the high-speed network accuracy evaluation factors, provides a detailed DDoS attack taxonomy, and classifies detection techniques. Moreover, the existing literature is inspected through a qualitative analysis, with respect to the factors extracted from the presented taxonomy of irregular traffic pattern detection. Different research directions are suggested to support researchers in identifying and designing the optimal solution by highlighting the issues and challenges of DDoS attacks on high-speed networks.

show abstract

“…The simulation cloud platform is based on the establishment of Spark framework, which is used for the unified analysis engine of large-scale data processing to set up multiple servers as cloud service clusters. There are two virtual nodes in each cloud node as slave nodes [11]. The experimental environment configuration is displayed in Table 1.…”

Section: Simulation Platform Environmentmentioning

confidence: 99%

Design of power grid full-service data monitoring platform based on cloud computing

Wei

Zhang

Duan

et al. 2023

Second International Conference on Algorithms, Microchips, and Network Applications (AMNA 2023)

View full text Add to dashboard Cite

The use of power monitoring and management system in power enterprises has significantly improved the convenience, fault tolerance, and related economic income of power grid in power supply. At present, there are still many problems in the management system, such as the lack of unified planning. Many functions related to power monitoring and management often need a lot of software to be repeatedly installed in the system to complete the functions required by customers. Meanwhile, this software often has duplicate functions. Power data cannot be shared, and the upgrade of the system will encounter many obstacles. To meet the data communication requirements of component-based power monitoring and management system, this paper applies the characteristics of multi-point distribution and real-time and efficient transmission of data based on model-based information transmission. It also applies component-based data monitoring technology to complete the design. For the power data content of monitoring and calculation, a reusable framework is applied to facilitate user expansion. Users can complete the interface layout by dragging controls on the control interface of the power monitoring and management system, and then realize data customization and saving by binding the database. Finally, the monitoring of the power grid data platforms under cloud service is realized, which is conducive to the effective and timely processing of blowout power grid data. It is of great significance to the safe and stable operation of the power system.

show abstract

RDMA-Based Apache Storm for High-Performance Stream Data Processing

Cited by 9 publications

References 20 publications

An Optimized RDMA QP Communication Mechanism for Hyperscale AI Infrastructure

An Optimized RDMA QP Communication Mechanism for Hyperscale AI Infrastructure

High-Speed Network DDoS Attack Detection: A Survey

Design of power grid full-service data monitoring platform based on cloud computing

Contact Info

Product

Resources

About