The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions

Abstract

Biological learning systems are outstanding in their ability to learn from limited training data compared to the most successful learning machines, i.e., Deep Neural Networks (DNNs). What are the key aspects that underlie this data efficiency gap is an unresolved question at the core of biological and artificial intelligence. We hypothesize that one important aspect is that biological systems rely on mechanisms such as foveations in order to reduce unnecessary input dimensions for the task at hand, e.g., background in object recognition, while state-of-the-art DNNs do not. Datasets to train DNNs often contain such unnecessary input dimensions, and these lead to more trainable parameters. Yet, it is not clear whether this affects the DNNs' data efficiency because DNNs are robust to increasing the number of parameters in the hidden layers, and it is uncertain whether this holds true for the input layer. In this paper, we investigate the impact of unnecessary input dimensions on the DNNs data efficiency, namely, the amount of examples needed to achieve certain generalization performance. Our results show that unnecessary input dimensions that are task-unrelated substantially degrade data efficiency. This highlights the need for mechanisms that remove task-unrelated dimensions, such as foveation for image classification, in order to enable data efficiency gains.

Breast Cancer Classification Using Equivariance Transition in Group Convolutional Neural Networks

“…To leverage the significance of DCT to fully exploit symmetry in mammography images, DCT was employed in this study to achieve rotational equivariance in the model's output representations. Table (1) provides a summary of the reviewed literature.…”

“…The past decade has witnessed wide and tremendous applications of Convolutional Neural Networks (CNNs) in the domain of computer vision, specifically medical image analysis, such as bone fracture detection [1]- [3], Pneumonia classification [4], Covid-19 detection [5]- [7], Breast cancer detection and segmentation [8]- [13], to mention but a few. Despite the recorded milestone, the scalability of CNNs is still subject to some controversies [14].…”

Smart Classroom Monitoring Using Novel Real-Time Facial Expression Recognition System

“…The research argues that analyzing facial expressions using a huge imagery dataset with unnecessary input dimensions can decrease the efficiency of the FER system. Similarly, another study concludes that images taken from different angles, low resolution, and noisy backgrounds can be problematic in automatic facial expression recognition [29][30][31][32][33]. Another research has argued that static images are not sufficient for automatic facial expression recognition and the authors conducted a study using recorded video of the classroom to improve the accuracy of FER [34][35][36][37].…”