Revisiting Random Channel Pruning for Neural Network Compression

Abstract

Channel (or 3D filter) pruning serves as an effective way to accelerate the inference of neural networks. There has been a flurry of algorithms that try to solve this practical problem, each being claimed effective in some ways. Yet, a benchmark to compare those algorithms directly is lacking, mainly due to the complexity of the algorithms and some custom settings such as the particular network configuration or training procedure. A fair benchmark is important for the further development of channel pruning.Meanwhile, recent investigations reveal that the channel configurations discovered by pruning algorithms are at least as important as the pre-trained weights. This gives channel pruning a new role, namely searching the optimal channel configuration. In this paper, we try to determine the channel configuration of the pruned models by random search. The proposed approach provides a new way to compare different methods, namely how well they behave compared with random pruning. We show that this simple strategy works quite well compared with other channel pruning methods. We also show that under this setting, there are surprisingly no clear winners among different channel importance evaluation methods, which then may tilt the research efforts into advanced channel configuration searching methods. Code will be released at https: //github.com/ofsoundof/random_channel_ pruning.

Rewarded meta-pruning: Meta Learning with Rewards for Channel Pruning

“…It can be inferred that this method has a lower error rate than every method, and this is achieved while keeping the FLOPs relatively low. The FLOPs, as compared to the baseline network [33], reduced by 52.55%, but the error has only increased by 0.84%. When compared with standard random pruning, for a similar reduction in FLOPs, there is a 0.63% lower error.…”

“…Baseline [33] 23.40% -4110M GAL-0.5 [37] 28.05% 9.06% 2341M SSS [25] 28.18% 9.21% 2341M HRank [36] 25.02% 7.67% 2311M Random Pruning [33] 24.87% 7.48% 2013M AutoPruner [43] 25.24% 7.85% 2005M Adapt-DCP [38] 24.85% 7.70% 1955M MetaPruning [40] 24.60% -2005M Rewarded meta-pruning 24.24% 7.35% 1950M…”

“…Baseline [33] 28.12% 9.71% 314M 0.75 MobileNetV2 [33] 30.20% -220M Random Pruning [10] 29.10% -223M AMC [18] 29.20% -220M MetaPruning [40] 28.80% -227M Greedy Selection [61] 28.80% -201M Adapt-DCP [38] 28.55% -216M Rewarded meta-pruning 28.51% 10.65% 199M Table 2 compares the performance of the Rewarded meta-pruning method with the state-of-the-art methods. The Rewarded meta-pruning method has a lower FLOPs than any other methods while showing only 0.39% higher error than the baseline.…”

“…An element is randomly picked from one of the two chosen genes for each index. Channel configurations in a local region of the configuration space tend to have similar metrics [33], so the new candidates also have similar accuracy and FLOPs. This makes the reward of at least the best candidate tend not to decrease.…”

Compact Image-Style Transfer: Channel Pruning on the Single Training of a Network

“…A study for random channel pruning [ 31 ] has been proposed to benchmark the channel pruning methods so far. This study showed that repeating randomly selecting partial channels during the training phase achieved the best result.…”