DO-Conv: Depthwise Over-Parameterized Convolutional Layer

Cao, Jinming; Li, Yangyan; Sun, Mingchao; Chen, Ying; Lischinski, Dani; Cohen–Or, Daniel; Chen, Baoquan; Tu, Changhe

doi:10.1109/tip.2022.3175432

Cited by 124 publications

(48 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Five main modifications are considered in the network: (1) replace the residual blocks (written as ResBlocks) with residual dense blocks; (2) replace Relu with LeakeyRelu; (3) use depthwise over-parameterized convolution (Do-conv) [ 33 ] instead of non-1 × 1 convolutional layer; (4) replace the original feature attention module (named FAM in MIMO-Unet) with the proposed EFAM; (5) replace the original asymmetric feature fusion (named AFF in MIMO-Unet) by MFFB in the shallow features.…”

Section: Methodsmentioning

confidence: 99%

“…Depthwise over-parameterized convolution Do-Conv has shown great potential in computer vision tasks [ 33 ]. In our network, Do-Conv is used to replace all non-1 × 1 convolution.…”

Section: Methodsmentioning

confidence: 99%

“…In Section 5, we test the trade-off between the number of parameters and the effect of correction. Depthwise over-parameterized convolution Do-Conv has shown great potential in computer vision tasks [33]. In our network, Do-Conv is used to replace all non-1 × 1 convolution.…”

Section: Network Architecturementioning

confidence: 99%

See 2 more Smart Citations

Multiscale Dense U-Net: A Fast Correction Method for Thermal Drift Artifacts in Laboratory NanoCT Scans of Semi-Conductor Chips

Liu

Han

et al. 2022

Entropy

View full text Add to dashboard Cite

The resolution of 3D structure reconstructed by laboratory nanoCT is often affected by changes in ambient temperature. Although correction methods based on projection alignment have been widely used, they are time-consuming and complex. Especially in piecewise samples (e.g., chips), the existing methods are semi-automatic because the projections lose attenuation information at some rotation angles. Herein, we propose a fast correction method that directly processes the reconstructed slices. Thus, the limitations of the existing methods are addressed. The method is named multiscale dense U-Net (MD-Unet), which is based on MIMO-Unet and achieves state-of-the-art artifacts correction performance in nanoCT. Experiments show that MD-Unet can significantly boost the correction performance (e.g., with three orders of magnitude improvement in correction speed compared with traditional methods), and MD-Unet+ improves 0.92 dB compared with MIMO-Unet in the chip dataset.

show abstract

Section: Methodsmentioning

confidence: 99%

“…Depthwise over-parameterized convolution Do-Conv has shown great potential in computer vision tasks [ 33 ]. In our network, Do-Conv is used to replace all non-1 × 1 convolution.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Multiscale Dense U-Net: A Fast Correction Method for Thermal Drift Artifacts in Laboratory NanoCT Scans of Semi-Conductor Chips

Liu

Han

et al. 2022

Entropy

View full text Add to dashboard Cite

show abstract

“…It is generally believed that the increase of network depth will improve network expression ability. Inspired by the literature [51], we employ an over‐parameterized convolutional layer, i.e. a conventional layer with additional depthwise convolution.…”

Section: Proposed Methodsmentioning

confidence: 99%

Learning compact ConvNets through filter pruning based on the saliency of a feature map

Liu

et al. 2021

IET Image Processing

View full text Add to dashboard Cite

With the performance increase of convolutional neural network (CNN), the disadvantages of CNN's high storage and high power consumption are followed. Among the methods mentioned in various literature, filter pruning is a crucial method for constructing lightweight networks. However, the current filter pruning method is still challenged by complicated processes and training inefficiency. This paper proposes an effective filter pruning method, which uses the saliency of the feature map (SFM), i.e. information entropy, as a theoretical guide for whether the filter is essential. The pruning principle use here is that the filter with a weak saliency feature map in the early stage will not significantly improve the final accuracy. Thus, one can efficiently prune the non-salient feature map with a smaller information entropy and the corresponding filter. Besides, an over-parameterized convolution method is employed to improve the pruned model's accuracy without increasing parameter at inference time. Experimental results show that without introducing any additional constraints, the effectiveness of this method in FLOPs and parameters reduction with similar accuracy has advanced the state-of-the-art. For example, on CIFAR-10, the pruned VGG-16 achieves only a small loss of 0.39% in Top-1 accuracy with a factor of 83.3% parameters, and 66.7% FLOPs reductions. On ImageNet-100, the pruned ResNet-50 achieves only a small accuracy degradation of 0.76% in Top-1 accuracy with a factor of 61.19% parameters, and 62.98% FLOPs reductions.

show abstract

“…The benefit of linear overparameterization in accelerating the training of deep neural networks and improving accuracy has received considerable attention in recent works [3,11,14,15,22,57]. Several of these previous works propose overparameterizing a convolutional layer during training by using a series of linear convolutional layers.…”

Section: Related Workmentioning

confidence: 99%

Restructurable Activation Networks

Bhardwaj¹,

Ward²,

Tung³

et al. 2022

Preprint

View full text Add to dashboard Cite

Is it possible to restructure the non-linear activation functions in a deep network to create hardwareefficient models? To address this question, we propose a new paradigm called Restructurable Activation Networks (RANs) that manipulate the amount of non-linearity in models to improve their hardware-awareness and efficiency. First, we propose RAN-explicit (RAN-e) -a new hardwareaware search space and a semi-automatic search algorithm -to replace inefficient blocks with hardware-aware blocks. Next, we propose a training-free model scaling method called RAN-implicit (RAN-i) where we theoretically prove the link between network topology and its expressivity in terms of number of non-linear units. We demonstrate that our networks achieve state-of-the-art results on ImageNet at different scales and for several types of hardware. For example, compared to EfficientNet-Lite-B0, RANe achieves a similar accuracy while improving Frames-Per-Second (FPS) by 1.5× on Arm micro-NPUs. On the other hand, RAN-i demonstrates up to 2× reduction in #MACs over ConvNexts with a similar or better accuracy. We also show that RAN-i achieves nearly 40% higher FPS than ConvNext on Arm-based datacenter CPUs. Finally, RAN-i based object detection networks achieve a similar or higher mAP and up to 33% higher FPS on datacenter CPUs compared to ConvNext based models.

show abstract

DO-Conv: Depthwise Over-Parameterized Convolutional Layer

Cited by 124 publications

References 21 publications

Multiscale Dense U-Net: A Fast Correction Method for Thermal Drift Artifacts in Laboratory NanoCT Scans of Semi-Conductor Chips

Multiscale Dense U-Net: A Fast Correction Method for Thermal Drift Artifacts in Laboratory NanoCT Scans of Semi-Conductor Chips

Learning compact ConvNets through filter pruning based on the saliency of a feature map

Restructurable Activation Networks

Contact Info

Product

Resources

About