SCANN: Synthesis of Compact and Accurate Neural Networks

Hassantabar, Shayan; Wang, Zeyu; Jha, Niraj K.

doi:10.48550/arxiv.1904.09090

Cited by 8 publications

(22 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[22], [23], [24] propose to grow part of connections based on the gradients of loss and prune it back to the desired sparsity. Similar efforts with different growing criterion are made in [25], [26], [27]. While all the non-zero weights in the existing grow-and-prune works are free to be updated, our work fundamentally differs in that the tiny sub-model is permanently frozen during all processes to guarantee the weight hierarchy.…”

Section: Related Workmentioning

confidence: 99%

Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting

Qin¹,

Zhang²,

Sun³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) have shown to provide superb performance in many real life applications, but their large computation cost and storage requirement have prevented them from being deployed to many edge and internet-of-things (IoT) devices. Sparse deep neural networks, whose majority weight parameters are zeros, can substantially reduce the computation complexity and memory consumption of the models. In real-use scenarios, devices may suffer from large fluctuations of the available computation and memory resources under different environment, and the quality of service (QoS) is difficult to maintain due to the long tail inferences with large latency. Facing the real-life challenges, we propose to train a sparse model that supports multiple sparse levels. That is, a hierarchical structure of weights are satisfied such that the locations and the values of the non-zero parameters of the more-sparse sub-model are a subset of the less-sparse sub-model. In this way, one can dynamically select the appropriate sparsity level during inference, while the storage cost is capped by the least sparse sub-model. We have verified our methodologies on a variety of DNN models and tasks, including the ResNet-50, PointNet++, GNMT, and graph attention networks. We obtain sparse sub-models with an average of 13.38% weights and 14.97% FLOPs, while the accuracies are as good as their dense counterparts. More-sparse sub-models with 5.38% weights and 4.47% of FLOPs, which are subsets of the less-sparse ones, can be obtained with only 3.25% relative accuracy loss. In addition, our proposed hierarchical model structure supports the mechanism to inference the first part of the model with less sparsity, and dynamically reroute to the more-sparse level if the real-time latency constraint is estimated to be violated. Preliminary analysis shows that we can improve the QoS by one or two nines depending on the task and the computation-memory resources of the inference engine.

show abstract

Section: Related Workmentioning

confidence: 99%

Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting

Qin¹,

Zhang²,

Sun³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Han et al [32] have shown the effectiveness of pruning in removing redundancy in CNNs and multilayer-perceptron architectures. Grow-andprune DNN synthesis uses network growth followed by network pruning in an iterative process to improve model performance while ensuring its compactness [14], [15].…”

Section: Efficient Neural Network Synthesismentioning

confidence: 99%

“…Hence, MHDeep uses synthetic data drawn from the same probability distribution as real data to augment the dataset. It also leverages a grow-andprune DNN synthesis approach [14], [15] to train accurate and computationally efficient neural network models to detect the mental health condition of the user.…”

Section: Introductionmentioning

confidence: 99%

MHDeep: Mental Health Disorder Detection System based on Body-Area and Deep Neural Networks

Hassantabar,

Zhang,

Yin

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Mental health problems impact quality of life of millions of people around the world. However, diagnosis of mental health disorders is a challenging problem that often relies on self-reporting by patients about their behavioral patterns and social interactions. Therefore, there is a need for new strategies for diagnosis and daily monitoring of mental health conditions. The recent introduction of body-area networks consisting of a plethora of accurate sensors embedded in smartwatches and smartphones and edge-compatible deep neural networks (DNNs) points towards a possible solution. Such wearable medical sensors (WMSs) enable continuous monitoring of physiological signals in a passive and non-invasive manner. However, disease diagnosis based on WMSs and DNNs, and their deployment on edge devices, such as smartphones, remains a challenging problem. To this end, we propose a framework called MHDeep that utilizes commercially available WMSs and efficient DNN models to diagnose three important mental health disorders: schizoaffective, major depressive, and bipolar. MHDeep uses eight different categories of data obtained from sensors integrated in a smartwatch and smartphone. These categories include various physiological signals and additional information on motion patterns and environmental variables related to the wearer. MHDeep eliminates the need for manual feature engineering by directly operating on the data streams obtained from participants. Since the amount of the data is limited, MHDeep uses a synthetic data generation module to augment real data with synthetic data drawn from the same probability distribution. We use the synthetic dataset to pre-train the weights of the DNN models, thus imposing a prior on the weights. We use a grow-and-prune DNN synthesis approach to learn both the architecture and weights during the training process. We use three different data partitions to evaluate the MHDeep models trained with data collected from 74 individuals. We conduct two types of evaluations: at the data instance level and at the patient level. MHDeep achieves an average test accuracy, across the three data partitions, of 90.4%, 87.3%, and 82.4%, respectively, for classifications between healthy and schizoaffective disorder instances, healthy and major depressive disorder instances, and healthy and bipolar disorder instances. At the patient level, MHDeep DNNs achieve an accuracy of 100%, 100%, and 90.0% for the three mental health disorders, respectively, based on inference that uses 40, 16, and 22 minutes of data from each patient.

show abstract

“…Since the dataset is very small, consisting of just 140 datapoints, it prevented us from being able to adequately train a neural network [52]. However, when our methodology is applied to a larger scope of cyberattacks, a neural network model might be an effective tool [53].…”

Section: Applying Machine Learningmentioning

confidence: 99%

SHARKS: Smart Hacking Approaches for RisK Scanning in Internet-of-Things and Cyber-Physical Systems based on Machine Learning

Saha,

Aaraj,

Ajjarapu

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Cyber-physical systems (CPS) and Internet-of-Things (IoT) devices are increasingly being deployed across multiple functionalities, ranging from healthcare devices and wearables to critical infrastructures, e.g., nuclear power plants, autonomous vehicles, smart cities, and smart homes. These devices are inherently not secure across their comprehensive software, hardware, and network stacks, thus presenting a large attack surface that can be exploited by hackers. In this article, we present an innovative technique for detecting unknown system vulnerabilities, managing these vulnerabilities, and improving incident response when such vulnerabilities are exploited. The novelty of this approach lies in extracting intelligence from known real-world CPS/IoT attacks, representing them in the form of regular expressions, and employing machine learning (ML) techniques on this ensemble of regular expressions to generate new attack vectors and security vulnerabilities. Our results show that 10 new attack vectors and 122 new vulnerability exploits can be successfully generated that have the potential to exploit a CPS or an IoT ecosystem. The ML methodology achieves an accuracy of 97.4% and enables us to predict these attacks efficiently with an 87.2% reduction in the search space. We demonstrate the application of our method to the hacking of the in-vehicle network of a connected car. To defend against the known attacks and possible novel exploits, we discuss a defense-in-depth mechanism for various classes of attacks and the classification of data targeted by such attacks. This defense mechanism optimizes the cost of security measures based on the sensitivity of the protected resource, thus incentivizing its adoption in real-world CPS/IoT by cybersecurity practitioners.

show abstract

SCANN: Synthesis of Compact and Accurate Neural Networks

Cited by 8 publications

References 0 publications

Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting

Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting

MHDeep: Mental Health Disorder Detection System based on Body-Area and Deep Neural Networks

SHARKS: Smart Hacking Approaches for RisK Scanning in Internet-of-Things and Cyber-Physical Systems based on Machine Learning

Contact Info

Product

Resources

About