TelaMalloc: Efficient On-Chip Memory Allocation for Production Machine Learning Accelerators

Maas, Martin; Beaugnon, Ulysse; Chauhan, Arun; Ilbeyi, Berkin

doi:10.1145/3567955.3567961

Cited by 11 publications

(1 citation statement)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Works such as [35,51] have designed various heuristic memory planning algorithms for static DNNs only. Tela-Malloc [43] performs memory management on the fly for static control-flow graphs with known intermediate tensor shapes and sizes. It does not fully consider the DNN control-flow dynamism and dynamic shapes.…”

Section: Related Workmentioning

confidence: 99%

SoD ² : Statically Optimizing Dynamic Deep Neural Network Execution

Niu,

Agrawal,

Ren

2024

Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems,

View full text Add to dashboard Cite

Though many compilation and runtime systems have been developed for DNNs in recent years, the focus has largely been on static DNNs. Dynamic DNNs, where tensor shapes and sizes and even the set of operators used are dependent upon the input and/or execution, are becoming common. This paper presents SoD 2 , a comprehensive framework for optimizing Dynamic DNNs. The basis of our approach is a classification of common operators that form DNNs, and the use of this classification towards a Rank and Dimension Propagation (RDP) method. This framework statically determines the shapes of operators as known constants, symbolic constants, or operations on these. Next, using RDP we enable a series of optimizations, like fused code generation, execution (order) planning, and even runtime memory allocation plan generation. By evaluating the framework on 10 emerging Dynamic DNNs and comparing it against several existing systems, we demonstrate both reductions in execution latency and memory requirements, with RDP-enabled key optimizations responsible for much of the gains. Our evaluation results show that SoD 2 runs up to 3.9× faster than these systems while saving up to 88% peak memory consumption. CCS CONCEPTS• Computing methodologies → Neural networks; • Software and its engineering → Source code generation; • Human-centered computing → Mobile computing.

show abstract

Section: Related Workmentioning

confidence: 99%