“…The scale of modern deep learning systems, and the rich opportunities for commercial gain in the sector, has seen a steady drift of researchers and research breakthroughs from academia across to industry. 10 Figure 1 FDA approvals for devices incorporating AI Approvals by the US regulator the FDA of clinical systems incorporating artificial intelligence capabilities have increased dramatically over the past decade. (Source: US Food and Drug Administration, 2022).…”
“…The scale of modern deep learning systems, and the rich opportunities for commercial gain in the sector, has seen a steady drift of researchers and research breakthroughs from academia across to industry. 10 Figure 1 FDA approvals for devices incorporating AI Approvals by the US regulator the FDA of clinical systems incorporating artificial intelligence capabilities have increased dramatically over the past decade. (Source: US Food and Drug Administration, 2022).…”
“…Clearly, even with the lower estimates, if the current trends were to persist, the energy consumption in training alone will more than off-set the energy efficiency from geometrical scaling or that due to architectures. In fact, more recent analysis suggests that the computational requirements for training NLP models may be many more orders of magnitude higher than the range we have addressed [21][22][23][24].…”
Section: Machine Learning In Natural Language Processingmentioning
We examine the computational energy requirements of different systems driven by the geometrical scaling law (known as Moore's law or Dennard Scaling for geometry) and increasing use of Artificial Intelligence/ Machine Learning (AI/ML) over the last decade. With more scientific and technology applications based on data-driven discovery, machine learning methods, especially deep neural networks, have become widely used. In order to enable such applications, both hardware accelerators and advanced AI/ML methods have led to the introduction of new architectures, system designs, algorithms, and software. Our analysis of energy trends indicates three important observations: 1) Energy efficiency due to geometrical scaling is slowing down; 2) The energy efficiency at the bit-level does not translate into efficiency at the instruction-level, or at the systemlevel for a variety of systems, especially for large-scale AI/ML accelerators or supercomputers; 3) At the application level, general-purpose AI/ML methods can be computationally energy intensive, off-setting the gains in energy from geometrical scaling and special purpose accelerators. Further, our analysis provides specific pointers for integrating energy efficiency with performance analysis for enabling high-performance and sustainable computing in the future.
“…Compute Requirements Specifically with regard to statistical significance in Section 5, there is a stark tension between the hardware requirements of modern methods (Sevilla et al, 2022) and the computational budget of the average researcher as well as the uncertainty under which experimental results are interpreted. Significance tests require many runs to produce reliable results: Neural network performance may fluctuate wildly, 6 and thus pose daunting computational costs, which but the best-funded research labs can afford (Hooker, 2021).…”
The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, as with other fields employing DL techniques, there has been a lack of common experimental standards compared to more established disciplines. Starting from fundamental scientific principles, we distill ongoing discussions on experimental standards in DL into a single, widely-applicable methodology. Following these best practices is crucial to strengthening experimental evidence, improve reproducibility and enable scientific progress. These standards are further collected in a public repository to help them transparently adapt to future needs.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
