Interpreting Layered Neural Networks via Hierarchical Modular Representation

Watanabe, Chihiro

doi:10.1007/978-3-030-36802-9_40

Cited by 7 publications

(6 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some previous empirical studies found that neural networks showed modularity and community, so more researchers began empirical research based on community discovery and analysis. Some research work attempts to study the modularity and community structure of neural networks at the neuron level or sub-network level [22][23][24][25][26][27][28][29]. In other research work, the architecture with modular characteristics is trained through parameter isolation or regularization during training, or the degree of modularity is improved, so as to develop a neural network with better modularity [30][31][32][33].…”

Section: Empirical Research Methods For Community Detectionmentioning

confidence: 99%

Understanding the class separation process of convolutional neural networks

Yang,

Lian,

Liu

et al. 2023

ACE

View full text Add to dashboard Cite

Extensive research shows that deep neural networks (DNNs) capture better feature representations than the previous handcrafted feature engineering, which leads to a significant performance improvement. However, it raises the question of how does the neural network spontaneously learn the intermediate representation, during the training process, to correctly separate the test data set into different categories? In this paper, based on the existing research work, we continue to take a small step towards understanding the dynamics of convolutional neural networks (CNN). Specifically, we model the test data set as a relationship graph, and each intermediate layer representation learned by CNN will transform the relationship graph correspondingly. Through these continuous transformations, we investigate the class separation process of CNN from two perspectives: 1) show the evolution of the degree of correlation between samples through the histogram of similarity between vertices; 2) by visualizing and quantifying the degree of separation (i.e., modularity) of the important relationship subgraph, we can show the contribution of each convolution layer of CNN to class separation. In the preliminary experiment, it can be found that: 1) each convolution layer of CNN gradually reduced the similarity of most edges between the samples to a very low level, that is, the dense relationship graph gradually evolved into a sparse graph; 2) the remaining important relationship subgraph consist of edges with high similarity shows evident modularity; 3) both visualization and quantification results show that the modularity of intra-class subgraph increases when the layers go deeper. Moreover, the degradation and plateau in the modularity curve reveal the existence of redundant layers. In this paper, we reveal some more in-depth dynamics of CNN, and introduced another modularity, which is widely used as the theoretical guidance tool of layer pruning. It can save more model parameters without losing the accuracy of classification.

show abstract

Section: Empirical Research Methods For Community Detectionmentioning

confidence: 99%

Understanding the class separation process of convolutional neural networks

Yang,

Lian,

Liu

et al. 2023

ACE

View full text Add to dashboard Cite

show abstract

“…Our work is perhaps most closely related to a series of papers by Watanabe and colleagues (Watanabe et al, 2018;Watanabe, 2019;2020) in which trained networks are decomposed into clusters of "similar" units with the aim of understanding and simplifying those networks. They quantify the similarity of units using a combination of both incoming and outgoing weights.…”

Section: Related Workmentioning

confidence: 97%

Clustering units in neural networks: upstream vs downstream information

Lange¹,

Rolnick²,

Kording³

2022

Preprint

View full text Add to dashboard Cite

It has been hypothesized that some form of "modular" structure in artificial neural networks should be useful for learning, compositionality, and generalization. However, defining and quantifying modularity remains an open problem. We cast the problem of detecting functional modules into the problem of detecting clusters of similar-functioning units. This begs the question of what makes two units functionally similar. For this, we consider two broad families of methods: those that define similarity based on how units respond to structured variations in inputs ("upstream"), and those based on how variations in hidden unit activations affect outputs ("downstream"). We conduct an empirical study quantifying modularity of hidden layer representations of simple feedforward, fully connected networks, across a range of hyperparameters. For each model, we quantify pairwise associations between hidden units in each layer using a variety of both upstream and downstream measures, then cluster them by maximizing their "modularity score" using established tools from network science. We find two surprising results: first, dropout dramatically increased modularity, while other forms of weight regularization had more modest effects. Second, although we observe that there is usually good agreement about clusters within both upstream methods and downstream methods, there is little agreement about the cluster assignments across these two families of methods. This has important implications for representation-learning, as it suggests that finding modular representations that reflect structure in inputs (e.g. disentanglement) may be a distinct goal from learning modular representations that reflect structure in outputs (e.g. compositionality).

show abstract

“…These methods involve no data or runtime analysis. In contrast, [18], [42], [110], [143], [256] each perform partitioning and cluster analysis based on how neurons associate with inputs and/or outputs. In particular, [110] present a statistical pipeline for quantifying the interpretability of neuron clusters with no human in the loop.…”

Section: Modular Partitionings (Post Hoc)mentioning

confidence: 99%

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Tilman¹,

Ho²,

Casper³

et al. 2022

Preprint

View full text Add to dashboard Cite

The last decade of machine learning has seen drastic increases in scale and capabilities, and deep neural networks (DNNs) are increasingly being deployed across a wide range of domains. However, the inner workings of DNNs are generally difficult to understand, raising concerns about the safety of using these systems without a rigorous understanding of how they function. In this survey, we review literature on techniques for interpreting the inner components of DNNs, which we call inner interpretability methods. Specifically, we review methods for interpreting weights, neurons, subnetworks, and latent representations with a focus on how these techniques relate to the goal of designing safer, more trustworthy AI systems. We also highlight connections between interpretability and work in modularity, adversarial robustness, continual learning, network compression, and studying the human visual system. Finally, we discuss key challenges and argue for future work in interpretability for AI safety that focuses on diagnostics, benchmarking, and robustness.

show abstract

Interpreting Layered Neural Networks via Hierarchical Modular Representation

Cited by 7 publications

References 24 publications

Understanding the class separation process of convolutional neural networks

Understanding the class separation process of convolutional neural networks

Clustering units in neural networks: upstream vs downstream information

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Contact Info

Product

Resources

About