Synopsis of the PhD Thesis - Network Computations in Artificial Intelligence

Mocanu, Decebal Constantin

doi:10.1109/itc30.2018.00027

Cited by 6 publications

(5 citation statements)

References 61 publications

(70 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, while training them with a fixed sparse topology can lead to good performance [25], it is hard to find an optimal sparse topology to fit the data distribution before training. This problems was addressed by introducing the adaptive sparse connectivity concept through its first instantiation, the Sparse Evolutionary Training (SET) algorithm [24,26]. SET is a straightforward strategy that starts from random sparse networks and can achieve good performance based on magnitude weights pruning and regrowing after each training epoch.…”

Section: Sparse Neural Networkmentioning

confidence: 99%

Topological Insights into Sparse Neural Networks

Liu

Lee

Yaman

et al. 2021

Machine Learning and Knowledge Discovery in Databases

Self Cite

View full text Add to dashboard Cite

Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivity, has emerged to allow training sparse neural networks from scratch by optimizing the sparse structure during training. However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions. This comparison becomes increasingly complex as the number of possible topological comparisons increases exponentially with the size of networks. In this work, we introduce an approach to understand and compare sparse neural network topologies from the perspective of graph theory. We first propose Neural Network Sparse Topology Distance (NNSTD) to measure the distance between different sparse neural networks. Further, we demonstrate that sparse neural networks can outperform over-parameterized models in terms of performance, even without any further structure optimization. To the end, we also show that adaptive sparse connectivity can always unveil a plenitude of sparse sub-networks with very different topologies which outperform the dense model, by quantifying and comparing their topological evolutionary processes. The latter findings complement the Lottery Ticket Hypothesis by showing that there is a much more efficient and robust way to find "winning tickets". Altogether, our results start enabling a better theoretical understanding of sparse neural networks, and demonstrate the utility of using graph theory to analyze them.

show abstract

Section: Sparse Neural Networkmentioning

confidence: 99%

Topological Insights into Sparse Neural Networks

Liu

Lee

Yaman

et al. 2021

Machine Learning and Knowledge Discovery in Databases

Self Cite

View full text Add to dashboard Cite

show abstract

“…Dynamic sparse training (DST) is a sparse-to-sparse training paradigm to learn a sparse (small) DNN model based on a dense (big) one [19], [27]. Specifically, DST starts training with a sparse model structure initialized from a dense model.…”

Section: Dynamic Sparse Trainingmentioning

confidence: 99%

Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deployment

Zhu

Wang

Xiao

2022

Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

View full text Add to dashboard Cite

The size of deep learning models in artificial intelligence (AI) software is increasing rapidly, hindering the large-scale deployment on resource-restricted devices (e.g., smartphones). To mitigate this issue, AI software compression plays a crucial role, which aims to compress model size while keeping high performance. However, the intrinsic defects in a big model may be inherited by the compressed one. Such defects may be easily leveraged by adversaries, since a compressed model is usually deployed in a large number of devices without adequate protection. In this article, we aim to address the safe model compression problem from the perspective of safety-performance co-optimization. Specifically, inspired by the test-driven development (TDD) paradigm in software engineering, we propose a test-driven sparse training framework called SafeCompress. By simulating the attack mechanism as safety testing, SafeCompress can automatically compress a big model to a small one following the dynamic sparse training paradigm. Then, considering two kinds of representative and heterogeneous attack mechanisms, i.e., black-box membership inference attack and white-box membership inference attack, we develop two concrete instances called BMIA-SafeCompress and WMIA-SafeCompress. Further, we implement another instance called MMIA-SafeCompress by extending SafeCompress to defend against the occasion when adversaries conduct black-box and white-box membership inference attacks simultaneously. We conduct extensive experiments on five datasets for both computer vision and natural language processing tasks. The results show the effectiveness and generalizability of our framework. We also discuss how to adapt SafeCompress to other attacks besides membership inference attack, demonstrating the flexibility of SafeCompress.

show abstract

“…Mocanu et al [51] have trained sparse restricted Boltzmann machines that have fixed scale-free and small-world connectivity. After that, Mocanu et al [52,53] have introduced the sparse evolutionary training procedure and the concept of adaptive connectivity for intrinsically sparse networks to fit the data distribution. The Nest algorithm [13] gets rid of a fully connected network at the beginning by a grow-andprune paradigm, that is, expanding a small randomly initialized sparse network to a large one and then shrink it down.…”

Section: Intrinsically Sparse Neural Networkmentioning

confidence: 99%

“…Note that it is impossible to report also the accuracy for FC-MLPs as they cannot run on a typical laptop due to their very high memory and computational requirements. Moreover, even if it would be possible to run FC-MLP, this comparison is outside the scope of this paper and it would be redundant as it has been shown in [6,24,44,52,53,73] that SET-MLP typically outperforms its fully connected counterparts.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware

Liu

Mocanu

Matavalam

et al. 2020

Neural Comput & Applic

Self Cite

View full text Add to dashboard Cite

Artificial neural networks (ANNs) have emerged as hot topics in the research community. Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes. Particularly for microarray data, the very high dimensionality and the small number of samples make it difficult for machine learning techniques to handle. Furthermore, specialized hardware such as graphics processing unit (GPU) is expensive. Sparse neural networks are the leading approaches to address these challenges. However, off-the-shelf sparsity-inducing techniques either operate from a pretrained model or enforce the sparse structure via binary masks. The training efficiency of sparse neural networks cannot be obtained practically. In this paper, we introduce a technique allowing us to train truly sparse neural networks with fixed parameter count throughout training. Our experimental results demonstrate that our method can be applied directly to handle high-dimensional data, while achieving higher accuracy than the traditional two-phase approaches. Moreover, we have been able to create truly sparse multilayer perceptron models with over one million neurons and to train them on a typical laptop without GPU (https:// github.com/dcmocanu/sparse-evolutionary-artificial-neural-networks/tree/master/SET-MLP-Sparse-Python-Data-Struc tures), this being way beyond what is possible with any state-of-the-art technique.

show abstract

Synopsis of the PhD Thesis - Network Computations in Artificial Intelligence

Cited by 6 publications

References 61 publications

Topological Insights into Sparse Neural Networks

Topological Insights into Sparse Neural Networks

Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deployment

Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware

Contact Info

Product

Resources

About