Running Neural Networks on the NIC

Siracusano, Giuseppe; Galea, Salvator; Sanvito, Davide; Malekzadeh, Mohammad; Haddadi, Hamed; Antichi, Gianni; Bifulco, Roberto

doi:10.48550/arxiv.2009.02353

Cited by 8 publications

(15 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As for the complexity is concerned, we observe that system researchers [97]- [100] consider extremely simple models (with just 21 neurons [99] or 50 neurons [100] overall), whereas AI researchers train excessively big models (state of the art models compared in [92] employ in excess of hundredsthousands neurons per-class). Awareness of commercial-grade challenges and constraints helps landing commercial-grade models out of the lab, by explicitly parsimonious AI-model design (less than hundred thousands neurons for all 200 classes [93]) and optimized implementation (e.g., using domain specific accelerator and languages [101], [102]).…”

Section: A Efficiently Handling the Known (L1 To L2)mentioning

confidence: 99%

Landing AI on Networks: An equipment vendor viewpoint on Autonomous Driving Networks

Rossi,

Zhang

2022

Preprint

View full text Add to dashboard Cite

The tremendous achievements of Artificial Intelligence (AI) in computer vision, natural language processing, games and robotics, has extended the reach of the AI hype to other fields: in telecommunication networks, the long term vision is to let AI fully manage, and autonomously drive, all aspects of network operation. In this industry vision paper, we discuss challenges and opportunities of Autonomous Driving Network (ADN) driven by AI technologies. To understand how AI can be successfully landed in current and future networks, we start by outlining challenges that are specific to the networking domain, putting them in perspective with advances that AI has achieved in other fields. We then present a system view, clarifying how AI can be fitted in the network architecture. We finally discuss current achievements as well as future promises of AI in networks, mentioning a roadmap to avoid bumps in the road that leads to true large-scale deployment of AI technologies in networks.

show abstract

Section: A Efficiently Handling the Known (L1 To L2)mentioning

confidence: 99%

Landing AI on Networks: An equipment vendor viewpoint on Autonomous Driving Networks

Rossi,

Zhang

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In comparison, other traffic analytics have less stringent requirements. We operate traffic classification via a 1D-Convolutional Neural Network (CNN) model, which size (about 100 k weights) is smaller than typical 2D CNN models used for image processing, but is significantly larger than the toy-case models used in the related system work [39,40]. The model is equivalent to the one used in [12] trained with over 200 applications labels, which is about ten (four) times the typical (maximum) number of classes considered in the literature [10].…”

Section: Case Studymentioning

confidence: 99%

“…ASIC is used in [40] for DL inference at packet level but only on toy-models with 3 layers and 21 neurons, i.e., 5000× smaller than the model we use. Smart NIC is used in [39], that however limits model size to 50 binary neurons, i.e., 2000× fewer weights, each with a resolution 32× smaller than in our case study. To attain sub-microsecond latency, [39,40] restrict themselves to such tiny models that it becomes questionable if their execution can have any practical use given the significant distance of such shallow models from the depth needed embrace the expected benefits of DL.…”

Section: Related Workmentioning

confidence: 99%

“…Smart NIC is used in [39], that however limits model size to 50 binary neurons, i.e., 2000× fewer weights, each with a resolution 32× smaller than in our case study. To attain sub-microsecond latency, [39,40] restrict themselves to such tiny models that it becomes questionable if their execution can have any practical use given the significant distance of such shallow models from the depth needed embrace the expected benefits of DL. Our work takes the opposite viewpoint and tackles a timely and efficient execution of relevant DL models for edge intelligence by using the appropriate offloading hardware, i.e., TPUs.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

FENXI: Deep-learning Traffic Analytics at the Edge

Gallo,

Finamore,

Simon

et al. 2021

Preprint

View full text Add to dashboard Cite

Live traffic analysis at the first aggregation point in the ISP network enables the implementation of complex traffic engineering policies but is limited by the scarce processing capabilities, especially for Deep Learning (DL) based analytics. The introduction of specialized hardware accelerators i.e., Tensor Processing Unit (TPU), offers the opportunity to enhance processing capabilities of network devices at the edge. Yet, to date, no packet processing pipeline is capable of offering DL-based analysis capabilities in the data-plane, without interfering with network operations.In this paper, we present FENXI, a system to run complex analytics by leveraging TPU. The design of FENXI decouples forwarding operations and traffic analytics which operates at different granularities i.e., packet and flow levels. We conceive two independent modules that asynchronously communicate to exchange network data and analytics results, and design data structures to extract flow level statistics without impacting per-packet processing. We prototyped and evaluated FENXI on general-purpose servers considering both both adversarial and realistic network conditions. Our analysis shows that FENXI can sustains 100 Gbps line rate traffic processing requiring only limited resources, while also dynamically adapting to variable network conditions. CCS CONCEPTS• Hardware → Analysis and design of emerging devices and systems; • Networks → In-network processing.

show abstract

“…A few attempts have been made to run ML models within the network (top of Figure 1), as detailed in Table 4 and §9. The first class of works [40,[44][45][46], implemented binary neural networks on network interface cards (NICs), FPGA or in a software environment. Their attempts to implement on a switch-ASIC have failed both in scale and performance, as it is significantly more constrained in resources and functionality.…”

Section: Introductionmentioning

confidence: 99%

IIsy: Practical In-Network Classification

Zheng¹,

Xiong²,

Bui³

et al. 2022

Preprint

View full text Add to dashboard Cite

The rat race between user-generated data and data-processing systems is currently won by data. The increased use of machine learning leads to further increase in processing requirements, while data volume keeps growing. To win the race, machine learning needs to be applied to the data as it goes through the network. In-network classification of data can reduce the load on servers, reduce response time and increase scalability.In this paper, we introduce IIsy, implementing machine learning classification models in a hybrid fashion using offthe-shelf network devices. IIsy targets three main challenges of in-network classification: (i) mapping classification models to network devices (ii) extracting the required features and (iii) addressing resource and functionality constraints. IIsy supports a range of traditional and ensemble machine learning models, scaling independently of the number of stages in a switch pipeline. Moreover, we demonstrate the use of IIsy for hybrid classification, where a small model is implemented on a switch and a large model at the backend, achieving near optimal classification results , while significantly reducing latency and load on the servers.

show abstract

Running Neural Networks on the NIC

Cited by 8 publications

References 31 publications

Landing AI on Networks: An equipment vendor viewpoint on Autonomous Driving Networks

Landing AI on Networks: An equipment vendor viewpoint on Autonomous Driving Networks

FENXI: Deep-learning Traffic Analytics at the Edge

IIsy: Practical In-Network Classification

Contact Info

Product

Resources

About