A law of data separation in deep learning

Abstract

While deep learning has enabled significant advances in many areas of science, its black-box nature hinders architecture design for future artificial intelligence applications and interpretation for high-stakes decision-makings. We addressed this issue by studying the fundamental question of how deep neural networks process data in the intermediate layers. Our finding is a simple and quantitative law that governs how deep neural networks separate data according to class membership throughout all layers for classification. This law shows that each layer improves data separation at a constant geometric rate, and its emergence is observed in a collection of network architectures and datasets during training. This law offers practical guidelines for designing architectures, improving model robustness and out-of-sample performance, as well as interpreting the predictions.

Comparison of the Capacity of Several Machine Learning Tools to Assist Immunofluorescence-Based Detection of Anti-Neutrophil Cytoplasmic Antibodies

“…However, it would be an attractive prospect to use ML to improve the power of selected parameters. While neural networks are often compared to “black boxes”, theoretical effort is currently underway regarding “interpreting” their behavior [ 48 ]. These endeavors might in the future help improve biological intuition and thereby allow for substantial improvement of so-called hand-crafted features.…”