Non-Discriminative Data or Weak Model? On the Relative Importance of Data and Model Resolution

Abstract

We explore the question of how the resolution of the input image ("input resolution") affects the performance of a neural network when compared to the resolution of the hidden layers ("internal resolution"). Adjusting these characteristics is frequently used as a hyperparameter providing a trade-off between model performance and accuracy. An intuitive interpretation is that the reduced information content in the low-resolution input causes decay in the accuracy. In this paper, we show that up to a point, the input resolution alone plays little role in the network performance, and it is the internal resolution that is the critical driver of model quality. We then build on these insights to develop novel neural network architectures that we call Isometric Neural Networks. These models maintain a fixed internal resolution throughout their entire depth. We demonstrate that they lead to high accuracy models with low activation footprint and parameter count.

Patches Are All You Need?

“…A study predating vision transformers investigate isotropic (or "isometric") MobileNets (Sandler et al, 2019), and even implements patch embeddings under another name. Their architecture simply repeats an isotropic MobileNetv3 block.…”

“…In Figure 4 and 5, we visualize the (complete) weights of the patch embedding layers of a ConvMixer-1536/20 with p = 14 and a ConvMixer-768/32 with p = 7, respectively. Much like Sandler et al (2019), the layer consists of Gabor-like filters as well as "colorful globs" or rough edge detectors.…”

BasisNet: Two-stage Model Synthesis for Efficient Inference

“…the input image resolution As shown in Figure 8, even an extremely efficient lightweight model (MV3-small (0.35x128), computation overhead of 13.8M Madds) can lead to a performance boost from 77.7% to 78.9% (+1.2%). This experiment shows that resolution and multiplier can have an equivalent effect as reported in [21] and a lightweight model with a smaller computation overhead can bring most of the performance gain. Thus it might be more beneficial to scale the model multiplier and resolution coordinately [25].…”

Augmenting Convolutional networks with attention-based aggregation

“…As a downside, these pyramidal approaches dramatically reduce the resolution of the last layers, and hence the quality of their attention maps, making their predictions harder to interpret. Another shortcoming is their relatively high memory usage [50].…”