WeightNet: Revisiting the Design Space of Weight Networks

Ma, Ningning; Zhang, Xiangyu; Huang, Jiawei; Sun, Jian

doi:10.1007/978-3-030-58555-6_46

Cited by 83 publications

(53 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Meta networks (Munkhdalai and Yu, 2017) combine the parameters predicted by meta learner and the weights trained cross tasks making rapid generalization. The WeightNet (Ma et al, 2020) unifies the SENet (Hu et al, 2018) and CondConv (Yang et al, 2019), trains the network in the kernel space and takes the network to generate weights for the classification task.…”

Section: Meta Learning For Parameters Generalizationmentioning

confidence: 99%

Meta grayscale adaptive network for 3D integrated renal structures segmentation

Yang

et al. 2021

Medical Image Analysis

View full text Add to dashboard Cite

Three-dimensional (3D) integrated renal structures (IRS) segmentation targets segmenting the kidneys, renal tumors, arteries, and veins in one inference. Clinicians will benefit from the 3D IRS visual model for accurate preoperative planning and intraoperative guidance of laparoscopic partial nephrectomy (LPN). However, no success has been reported in 3D IRS segmentation due to the inherent challenges in grayscale distribution: low contrast caused by the narrow task-dependent distribution range of regions of interest (ROIs), and the networks representation preferences caused by the distribution variation inter-images. In this paper, we propose the Meta Greyscale Adaptive Network (MGANet), the first deep learning framework to simultaneously segment the kidney, renal tumors, arteries and veins on CTA images in one inference. It makes innovations in two collaborate aspects: 1) The Grayscale Interest Search (GIS) adaptively focuses segmentation networks on task-dependent grayscale distributions via scaling the window width and center with two cross-correlated coefficients for the first time, thus learning the fine-grained representation for fine segmentation. 2) The Meta Grayscale Adaptive (MGA) learning makes an image-level meta-learning strategy. It represents diverse robust features from multiple distributions, perceives the distribution characteristic, and generates the model parameters to fuse features dynamically according to image's distribution, thus adapting the grayscale distribution variation. This study enrolls 123 patients and the average Dice coefficients of the renal structures are up to 87.9%. Fine selection of the task-dependent grayscale distribution ranges and personalized fusion of multiple representations on different distributions will lead to better 3D IRS segmentation quality. Extensive experiments with promising results on renal structures reveal powerful segmentation accuracy and great clinical significance in renal cancer treatment.

show abstract

Section: Meta Learning For Parameters Generalizationmentioning

confidence: 99%

Meta grayscale adaptive network for 3D integrated renal structures segmentation

Yang

et al. 2021

Medical Image Analysis

View full text Add to dashboard Cite

show abstract

“…To solve this problem, dynamic filters are proposed one after another. One kind of dynamic filter [10,51,78,88] predicts coefficients to combine several expert filters which are then shared across all spatial pixels. Another kind of dynamic filter [7,39,42,58,66,67,83,92] generates spatial-specific filters.…”

Section: Related Workmentioning

confidence: 99%

ELSA: Enhanced Local Self-Attention for Vision Transformer

Zhou¹,

Wang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-attention is powerful in modeling long-range dependencies, but it is weak in local finer-level feature learning. The performance of local self-attention (LSA) is just on par with convolution and inferior to dynamic filters, which puzzles researchers on whether to use LSA or its counterparts, which one is better, and what makes LSA mediocre. To clarify these, we comprehensively investigate LSA and its counterparts from two sides: channel setting and spatial processing. We find that the devil lies in the generation and application of spatial attention, where relative position embeddings and the neighboring filter application are key factors. Based on these findings, we propose the enhanced local self-attention (ELSA) with Hadamard attention and the ghost head. Hadamard attention introduces the Hadamard product to efficiently generate attention in the neighboring case, while maintaining the high-order mapping. The ghost head combines attention maps with static matrices to increase channel capacity. Experiments demonstrate the effectiveness of ELSA. Without architecture / hyperparameter modification, drop-in replacing LSA with ELSA boosts Swin Transformer [48] by up to +1.4 on top-1 accuracy. ELSA also consistently benefits VOLO [83] from D1 to D5, where ELSA-VOLO-D5 achieves 87.2 on the ImageNet-1K without extra training images. In addition, we evaluate ELSA in downstream tasks. ELSA significantly improves the baseline by up to +1.9 box Ap / +1.3 mask Ap on the COCO, and by up to +1.9 mIoU on the ADE20K. Code is available at https://github.com/damo-cv/ELSA.

show abstract

“…Several recent works [40], [37], [41], [42] focus on "input dependent" dynamic inference, i.e., the path to inference is determined by the individual input, where the "easy" inputs go through simple paths and thus reduce computation on average. Our "dynamic rerouting" is fundamentally different in that ours is input independent, i.e., any input data can be switched from small to tiny sub-models from any designated switching point (e.g., 4 points for ResNet50).…”

Section: Comparison To Existing Dynamic Dnnsmentioning

confidence: 99%

“…Slimmable neural network [37] has one result that achieves 1.4% less accuracy than our tiny ResNet50 with 5× parameters (27.06% vs 5.43%). CondConv [41] or WeightNet [42] are even larger than the dense ResNet50.…”

Section: Comparison To Existing Dynamic Dnnsmentioning

confidence: 99%

Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting

Qin¹,

Zhang²,

Sun³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) have shown to provide superb performance in many real life applications, but their large computation cost and storage requirement have prevented them from being deployed to many edge and internet-of-things (IoT) devices. Sparse deep neural networks, whose majority weight parameters are zeros, can substantially reduce the computation complexity and memory consumption of the models. In real-use scenarios, devices may suffer from large fluctuations of the available computation and memory resources under different environment, and the quality of service (QoS) is difficult to maintain due to the long tail inferences with large latency. Facing the real-life challenges, we propose to train a sparse model that supports multiple sparse levels. That is, a hierarchical structure of weights are satisfied such that the locations and the values of the non-zero parameters of the more-sparse sub-model are a subset of the less-sparse sub-model. In this way, one can dynamically select the appropriate sparsity level during inference, while the storage cost is capped by the least sparse sub-model. We have verified our methodologies on a variety of DNN models and tasks, including the ResNet-50, PointNet++, GNMT, and graph attention networks. We obtain sparse sub-models with an average of 13.38% weights and 14.97% FLOPs, while the accuracies are as good as their dense counterparts. More-sparse sub-models with 5.38% weights and 4.47% of FLOPs, which are subsets of the less-sparse ones, can be obtained with only 3.25% relative accuracy loss. In addition, our proposed hierarchical model structure supports the mechanism to inference the first part of the model with less sparsity, and dynamically reroute to the more-sparse level if the real-time latency constraint is estimated to be violated. Preliminary analysis shows that we can improve the QoS by one or two nines depending on the task and the computation-memory resources of the inference engine.

show abstract

WeightNet: Revisiting the Design Space of Weight Networks

Cited by 83 publications

References 24 publications

Meta grayscale adaptive network for 3D integrated renal structures segmentation

Meta grayscale adaptive network for 3D integrated renal structures segmentation

ELSA: Enhanced Local Self-Attention for Vision Transformer

Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting

Contact Info

Product

Resources

About