SambaNova SN10 RDU:Accelerating Software 2.0 with Dataflow

Abstract

Monolithic large language models (LLMs) like GPT-4 have paved the way for modern generative AI applications. Training, serving, and maintaining monolithic LLMs at scale, however, remains prohibitively expensive and challenging. The disproportionate increase in computeto-memory ratio of modern AI accelerators have created a memory wall, necessitating new methods to deploy AI. Recent research has shown that a composition of many smaller expert models, each with several orders of magnitude fewer parameters, can match or exceed the capabilities of monolithic LLMs. Composition of Experts (CoE) is a modular approach that lowers the cost and complexity of training and serving. However, this approach presents two key challenges when using conventional hardware: (1) without fused operations, smaller models have lower operational intensity, which makes high utilization more challenging to achieve; and (2) hosting a large number of models can be either prohibitively expensive or slow when dynamically switching between them.In this paper, we describe how combining CoE, streaming dataflow, and a three-tier memory system scales the AI memory wall. We describe Samba-CoE, a CoE system with 150 experts and a trillion total parameters. We deploy Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) -a commercial dataflow accelerator architecture that has been codesigned for enterprise inference and training applications. The chip introduces a new three-tier memory system with on-chip distributed SRAM, on-package HBM, and off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out over multiple sockets. We demonstrate speedups ranging from 2× to 13× on various benchmarks running on eight RDU sockets compared with an unfused baseline. We show that for CoE inference deployments, the 8-socket RDU Node reduces machine footprint by up to 19×, speeds up model switching time by 15× to 31×, and achieves an overall speedup of 3.7× over a DGX H100 and 6.6× over a DGX A100.

AI and ML Accelerator Survey and Trends

“…• After last year releasing some impressive benchmark results for their reconfigurable AI accelerator technology [119] and this year publishing two deeper technology reveals [120], [121] and an applications paper with Argonne National Laboratory [122], SambaNova still has not provided any details from which we can estimate peak performance or power consumption of their solutions. • In May 2022, Intel's Habana Labs announced the second generations of the Goya inference accelerator and Gaudi training accelerator, named Greco and Gaudi2, respectively [123], [124].…”

ARK: Fully Homomorphic Encryption Accelerator with Runtime Data Generation and Inter-Operation Key Reuse

“…The time to load evk (r) rot s and plaintexts, which are used once per H-(I)DFT, becomes the hard bound of latency for H-(I)DFT. Even if we assume that integrating hundreds of MBs of on-chip memory is feasible with the current fabrication technology [43], [51], [66], we must fetch the single-use data of H-(I)DFT from off-chip memory due to its large aggregate size and limited data reuse opportunities. With the latest HBM3 [41], a system having a 3TB/s off-chip memory bandwidth is feasible [28], which would allow loading the single-use data in 2.1ms (resp., 0.2ms) for H-IDFT (H-DFT).…”

BTS

“…Recently, on-chip SRAM capacities have scaled significantly [6] such that the level of hundreds of MBs of on-chip SRAM is feasible, providing tens of TB/s of SRAM bandwidth [47,55,69]. While the bandwidth of the main-memory has also increased, its aggregate throughput is still more than an order of magnitude lower than the on-chip SRAM bandwidth [66], achieving a few TB/s of throughput even with high-bandwidth memory (HBM).…”