A Survey of Deep Learning for Scientific Discovery

Abstract

Over the past few years, we have seen fundamental breakthroughs in core problems in machine learning, largely driven by advances in deep neural networks. At the same time, the amount of data collected in a wide array of scientific domains is dramatically increasing in both size and complexity. Taken together, this suggests many exciting opportunities for deep learning applications in scientific settings. But a significant challenge to this is simply knowing where to start. The sheer breadth and diversity of different deep learning techniques makes it difficult to determine what scientific problems might be most amenable to these methods, or which specific combination of methods might offer the most promising first approach. In this survey, we focus on addressing this central issue, providing an overview of many widely used deep learning models, spanning visual, sequential and graph structured data, associated tasks and different training methods, along with techniques to use deep learning with less data and better interpret these complex models -two central considerations for many scientific use cases. We also include overviews of the full design process, implementation tips, and links to a plethora of tutorials, research summaries and open-sourced deep learning pipelines and pretrained models, developed by the community. We hope that this survey will help accelerate the use of deep learning across different scientific domains.

Social physics

“…There exists a large number of machine-learning and deep-learning surveys tailored to the needs of both specific [507,509,542,543,544,545] or general audiences [546,547,548,549,550,551,552,553]. Prioritising the latter, Ref.…”

“…Turning to the specifics of the AI, machine-learning, or deep-learning use in science, Ref. [544] overviews the techniques for applying deep-learning models in conjunction with limited data (self-supervision, semi-supervised learning, and data augmentation), as well as the techniques for interpretability and representation analyses. Ref.…”

Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads

“…Since the advent of Deep Learning (DL) as one of the most promising machine learning paradigms almost 10 years ago, deep neural networks have advanced the fields of computer vision, natural language processing, recommender systems, and gradually pervade an increasing number of scientific domains [1][2][3][4][5][6][7][8][9][10]. Due to the diverse nature of the problems under consideration, these DL workloads exhibit a wide range of computational characteristics and demands.…”

Learning the Travelling Salesperson Problem Requires Rethinking Generalization

“…In practice, the Concorde TSP solver [4] uses linear programming with carefully handcrafted heuristics to find solutions up to tens of thousands of nodes, but with prohibitive execution times. 2 Besides, the development of problem-specific OR solvers such as Concorde for novel or under-studied problems arising in scientific discovery [81,76] or computer architecture [67,21] requires significant time and specialized knowledge.…”