Artificial Neural Networks as Models of Robustness in Development and Regeneration: Stability of Memory During Morphological Remodeling

Hammelman, Jennifer; Lobo, Daniel; Levin, Michael

doi:10.1007/978-3-319-28495-8_3

Cited by 3 publications

(1 citation statement)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In fact, Glassman (1987) estimates that at least half of the neurons in the human brain fit under our definition of frivolous. The neural activity of DNNs for object recognition has been shown to resemble neural activity in parts of the brain (Yamins and DiCarlo 2016), and several works have aimed to understand redundancy and robustness in brains using artificial networks as models (Schuster 2008;Aerts et al 2016;Hammelman, Lobo, and Levin 2016). Thus, the framework presented in this paper could be helpful to understand prunability and redundancy in brains and how biological networks implicitly regularize.…”

Section: Discussionmentioning

confidence: 99%

Frivolous Units: Wider Networks Are Not Really That Wide

Casper

Boix

D’Amario

et al. 2021

AAAI

View full text Add to dashboard Cite

A remarkable characteristic of overparameterized deep neural networks (DNNs) is that their accuracy does not degrade when the network width is increased. Recent evidence suggests that developing compressible representations allows the complexity of large networks to be adjusted for the learning task at hand. However, these representations are poorly understood. A promising strand of research inspired from biology involves studying representations at the unit level as it offers a more granular interpretation of the neural mechanisms. In order to better understand what facilitates increases in width without decreases in accuracy, we ask: Are there mechanisms at the unit level by which networks control their effective complexity? If so, how do these depend on the architecture, dataset, and hyperparameters? We identify two distinct types of “frivolous” units that proliferate when the network’s width increases: prunable units which can be dropped out of the network without significant change to the output and redundant units whose activities can be expressed as a linear combination of others. These units imply complexity constraints as the function the network computes could be expressed without them. We also identify how the development of these units can be influenced by architecture and a number of training factors. Together, these results help to explain why the accuracy of DNNs does not degrade when width is increased and highlight the importance of frivolous units toward understanding implicit regularization in DNNs.

show abstract

Section: Discussionmentioning

confidence: 99%

Frivolous Units: Wider Networks Are Not Really That Wide

Casper

Boix

D’Amario

et al. 2021

AAAI

View full text Add to dashboard Cite

show abstract

The Biophysics of Regenerative Repair Suggests New Perspectives on Biological Causation

Levin

2020

BioEssays

Self Cite

View full text Add to dashboard Cite

Evolution exploits the physics of non-neural bioelectricity to implement anatomical homeostasis: a process in which embryonic patterning, remodeling, and regeneration achieve invariant anatomical outcomes despite external interventions. Linear "developmental pathways" are often inadequate explanations for dynamic large-scale pattern regulation, even when they accurately capture relationships between molecular components. Biophysical and computational aspects of collective cell activity toward a target morphology reveal interesting aspects of causation in biology. This is critical not only for unraveling evolutionary and developmental events, but also for the design of effective strategies for biomedical intervention. Bioelectrical controls of growth and form, including stochastic behavior in such circuits, highlight the need for the formulation of nuanced views of pathways, drivers of system-level outcomes, and modularity, borrowing from concepts in related disciplines such as cybernetics, control theory, computational neuroscience, and information theory. This approach has numerous practical implications for basic research and for applications in regenerative medicine and synthetic bioengineering.

show abstract