Sparsely Aggregated Convolutional Networks

Zhu, Lei; Deng, Ruizhi; Maire, Michael; Deng, Zhiwei; Mori, Greg; Tan, Ping

doi:10.1007/978-3-030-01258-8_12

Cited by 55 publications

(32 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Manually Designed Models Our search space contains many classic architectures designed by experts. To show the architecture discovered by our method is better than the others in the search space, we select two well-known architectures to compare with, namely U-Net [32] and SparseACN [48]. U-Net is a classic architecture that follows the encoder-decoder style.…”

Section: Resultsmentioning

confidence: 99%

SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

Zhang

Huang

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

In this paper, we aim at automatically searching an efficient network architecture for dense image prediction. Particularly, we follow the encoder-decoder style and focus on designing a connectivity structure for the decoder. To achieve that, we design a densely connected network with learnable connections, named Fully Dense Network, which contains a large set of possible final connectivity structures. We then employ gradient descent to search the optimal connectivity from the dense connections. The search process is guided by a novel loss function, which pushes the weight of each connection to be binary and the connections to be sparse. The discovered connectivity achieves competitive results on two segmentation datasets, while runs more than three times faster and requires less than half parameters compared to the state-of-the-art methods. An extensive experiment shows that the discovered connectivity is compatible with various backbones and generalizes well to other dense image prediction tasks. Code is available at https://github.com/wuhuikai/SparseMask.

show abstract

Section: Resultsmentioning

confidence: 99%

SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

Zhang

Huang

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…A wider 101-layer residual network also achieved higher accuracy on ImageNet classification than a 200-layer network with the same model complexity [38]. One possible explanation is that residual aggregation entangles outputs from each layer and thus hinders the ability to search for new features [42]. We hence implement inlayer shortcut connections for SeqConv with concatenation instead of addition to avoid such limitation.…”

Section: Related Workmentioning

confidence: 99%

“…The extra weights assigned to earlier features give rise to a vast number of required parameters growing at an asymptotic rate of O(n 2 ), where n is the width of the SeqConv layer, whereas a regular convolutional layer merely has a linear parameter growth rate. Recent study [42] suggests that this quadratic growth suffers from significant parameter redundancy. It is observed that DenseNet, which shares the same aggregation mechanism with SeqConv, has many skip connections with average absolute weights close to zero [42,13].…”

Section: Sequentially Aggregated Transformationsmentioning

confidence: 99%

See 1 more Smart Citation

Sequentially Aggregated Convolutional Networks

Huang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Modern deep networks generally implement a certain form of shortcut connections to alleviate optimization difficulties. However, we observe that such network topology alters the nature of deep networks. In many ways, these networks behave similarly to aggregated wide networks. We thus exploit the aggregation nature of shortcut connections at a finer architectural level and place them within wide convolutional layers. We end up with a sequentially aggregated convolutional (SeqConv) layer that combines the benefits of both wide and deep representations by aggregating features of various depths in sequence. The proposed SeqConv serves as a drop-in replacement of regular wide convolutional layers and thus could be handily integrated into any backbone network. We apply Seq-Conv to widely adopted backbones including ResNet and ResNeXt, and conduct experiments for image classification on public benchmark datasets. Our ResNet based network with a model size of ResNet-50 easily surpasses the performance of the 2.35× larger ResNet-152, while our ResNeXt based model sets a new state-of-the-art accuracy on ImageNet classification for networks with similar model complexity. The code and pre-trained models of our work are publicly available at https://github.com/ GroupOfAlchemists/SeqConv.

show abstract

“…Despite the success of both ResNets and DenseNets, both aggregation types have drawbacks. For ResNets, information from the outputs of shallower layers can be lost after multiple summations with deeper layer outputs (Zhu et al 2018). This restricts feature re-usage and limits feature exploration during training.…”

Section: Introductionmentioning

confidence: 99%

Deep Residual-Dense Lattice Network for Speech Enhancement

Nikzad

Nicolson

Gao

et al. 2020

AAAI

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) with residual links (ResNets) and causal dilated convolutional units have been the network of choice for deep learning approaches to speech enhancement. While residual links improve gradient flow during training, feature diminution of shallow layer outputs can occur due to repetitive summations with deeper layer outputs. One strategy to improve feature re-usage is to fuse both ResNets and densely connected CNNs (DenseNets). DenseNets, however, over-allocate parameters for feature re-usage. Motivated by this, we propose the residual-dense lattice network (RDL-Net), which is a new CNN for speech enhancement that employs both residual and dense aggregations without over-allocating parameters for feature re-usage. This is managed through the topology of the RDL blocks, which limit the number of outputs used for dense aggregations. Our extensive experimental investigation shows that RDL-Nets are able to achieve a higher speech enhancement performance than CNNs that employ residual and/or dense aggregations. RDL-Nets also use substantially fewer parameters and have a lower computational requirement. Furthermore, we demonstrate that RDL-Nets outperform many state-of-the-art deep learning approaches to speech enhancement. Availability: https://github.com/nick-nikzad/RDL-SE.

show abstract

Sparsely Aggregated Convolutional Networks

Cited by 55 publications

References 31 publications

SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

Sequentially Aggregated Convolutional Networks

Deep Residual-Dense Lattice Network for Speech Enhancement

Contact Info

Product

Resources

About