1.1 The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design

Abstract

The past decade has seen a remarkable series of advances in machine learning, and in particular deep learning approaches based on artificial neural networks, to improve our abilities to build more accurate systems across a broad range of areas, including computer vision, speech recognition, language translation, and natural language understanding tasks. This paper is a companion paper to a keynote talk at the 2020 International Solid-State Circuits Conference (ISSCC) discussing some of the advances in machine learning, and their implications on the kinds of computational devices we need to build, especially in the post-Moore's Law-era. It also discusses some of the ways that machine learning may also be able to help with some aspects of the circuit design process. Finally, it provides a sketch of at least one interesting direction towards much larger-scale multi-task models that are sparsely activated and employ much more dynamic, example-and task-based routing than the machine learning models of today.

On EDA solutions for reconfigurable memory-centric AI edge applications

“…This work aims at an intelligent edge device which supports different applications under diverse scenarios [5]. The device might employ a large capacity machine learning model, but the model will be sparsely-activated [13].…”

“…It is expected that the next generation smart edge systems will support various intelligent applications by employing multi-task machine learning models which would be dynamically activated [5]. Moreover, the system could implement various IMC technologies, such as SRAM [3], ReRAM [4], or near DRAM computing [8].…”

Analysis of Posit and Bfloat Arithmetic of Real Numbers for Machine Learning

“…In addition to hardware optimization for a certain class of tasks, using bfloats helps to move to half-precision numbers (in case of basic 16-bit implementation). This solution was applied in paper [13]. Unlike 16-bit numbers of the IEEE 754 standard, which can represent values in the range from to , bfloat16 covers the interval between and , while classical floats can only achieve that when using 32 bits.…”

An Overview of Machine Learning within Embedded and Mobile Devices–Optimizations and Applications

“…Besides, the recent breakthroughs of deep learning models in application areas such as computer vision [ 3 , 4 , 5 , 6 , 7 ], speech recognition [ 8 , 9 ], language translation, and processing [ 10 , 11 ], robotics, and healthcare [ 12 ] make this overlap a key research direction for the development of next-generation embedded devices. Thus, this has opened a research thrust between embedded devices and machine learning models termed “Embedded Machine Learning” where machine learning models are executed within resource-constrained environments [ 13 ]. This research surveys key issues within this convergence of embedded systems and machine learning.…”

“…General-purpose CPUs, even with their architectural modification over the years, including pipelining, deep cache memory hierarchies, multicore enhancements, etc., cannot meet the high computational demand of deep learning models. However, graphic processing units (GPUs), due to their high floating-point performance and thread-level parallelism, are more suitable for training deep learning models [ 13 ]. Extensive research is actively being carried out to develop suitable hardware acceleration units using FPGAs [ 20 , 21 , 22 , 23 , 24 , 25 , 26 ], GPUs, ASICs, and TPUs to create heterogeneous and sometimes distributed systems to meet up the high computational demand of deep learning models.…”