PSigmoid: Improving squeeze-and-excitation block with parametric sigmoid

Yao, Ying; Zhang, Nengbo; Shan, Peng; Miao, Ligang; Sun, Peng; Peng, Silong

doi:10.1007/s10489-021-02247-z

Cited by 15 publications

(5 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The σ and δ are activation functions. W 1 ∈ R f × f r , and W 2 ∈ f r × f represent two fully connected layers, where the r is reduction ratio to control capacity and computational cost [43].…”

Section: Excitationmentioning

confidence: 99%

A split-federated learning and edge-cloud based efficient and privacy-preserving large-scale item recommendation model

et al. 2023

View full text Add to dashboard Cite

The combination of federated learning and recommender system aims to solve the privacy problems of recommendation through keeping user data locally at the client device during the model training session. However, most existing approaches rely on user devices to fully compute the deep model designed for the large-scale item recommendation; therefore, imposing high calculation and communication overheads on resource-constrained user devices. Consequently, achieving efficient federated recommendations across ubiquitous mobile devices remains an open research problem. To this end, in this paper we propose an efficient and privacy-preserving federated learning framework which is based on the cloud-edge collaboration for large-scale item recommendation called SpFedRec. In our method, to reduce the computation and communication cost of the federated two-tower model, a split learning approach is applied to migrate the item model from participants’ edge devices to the computationally powerful cloud side and compress item data while transmitting. Meanwhile, to enhance the feature representation, the Squeeze-and-Excitation network mechanism is used on the backbone model to optimize the perception of dominant features. Moreover, because the gradients transmitted contain private information about the user; therefore, we propose a multi-party circular secret-sharing chain based on secret sharing for better privacy protection. Extensive experiments using plausible assumptions on two real-world datasets demonstrate that our proposed method improves the average computation time and communication cost by 23% and 49%, respectively. Furthermore, the proposed model accomplishes comparable performance with other state-of-art federated recommendation models.

show abstract

Section: Excitationmentioning

confidence: 99%

A split-federated learning and edge-cloud based efficient and privacy-preserving large-scale item recommendation model

et al. 2023

View full text Add to dashboard Cite

show abstract

“…An adaptive variant of logistic sigmoid named parametric sigmoid (psigmoid) 57 was proposed in [463,464]. 58 Similarly as in generalized hyperbolic tangent, it introduces two scaling parameters to a logistic sigmoid:…”

Section: Parametric Sigmoid (Psigmoid)mentioning

confidence: 99%

“…where a i is a trainable parameter for each neuron or channel i and b is a global trainable parameter [463].…”

Section: Parametric Sigmoid (Psigmoid)mentioning

confidence: 99%

See 1 more Smart Citation

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Kunc

Kléma²

2019

Preprint

View full text Add to dashboard Cite

Motivation: Gene expression profiling was made cheaper by the NIH LINCS program that profiles only ∼1, 000 selected landmark genes and uses them to reconstruct the whole profile. The D-GEX method employs neural networks to infer the whole profile. However, the original D-GEX can be further significantly improved. Results: We have analyzed the D-GEX method and determined that the inference can be improved using a logistic sigmoid activation function instead of the hyperbolic tangent. Moreover, we propose a novel transformative adaptive activation function that improves the gene expression inference even further and which generalizes several existing adaptive activation functions. Our improved neural network achieves average mean absolute error of 0.1340 which is a significant improvement over our reimplementation of the original D-GEX which achieves average mean absolute error 0.1637

show abstract

“…In the denoising task, each noise point is given weight, the low weight noise points are removed automatically, and the high weight noise points are retained. During this process, the network running efficiency can be improved, the parameters and computational cost can be reduced, and the recognition accuracy is improved [31]. As shown in Figure 4, by processing the feature map of convolutional, a one-dimensional vector with the same number of channels is obtained as the evaluation score of each channel [32], and then, the score is used for the corresponding channel to get the result.…”

mentioning

confidence: 99%

A Novel Deep Convolutional Neural Network Based on ResNet‐18 and Transfer Learning for Detection of Wood Knot Defects

et al. 2021

View full text Add to dashboard Cite

Wood defects are quickly identified from an optical image based on deep learning methodology, which effectively improves wood utilization. Traditional neural network techniques have not yet been employed for wood defect detection due to long training time, low recognition accuracy, and nonautomatical extraction of defect image features. In this work, a model (so-called ReSENet-18) for wood knot defect detection that combined deep learning and transfer learning is proposed. The “squeeze-and-excitation” (SE) module is firstly embedded into the “residual basic block” structure for a “SE-Basic-Block” module construction. This model has the advantages of the features that are extracted in the channel dimension, and it is fused in multiscale with original features. Instantaneously, the fully connected layer is replaced with a global average pooling; consequently, the model parameters could be reduced effectively. The experimental results show that the accuracy has reached 99.02%, meanwhile the training time is also reduced. It shows that the proposed deep convolutional neural network based on ReSENet-18 combined with transfer learning can improve the accuracy of defect recognition and has a potential application in the detection of wood knot defects.

show abstract

PSigmoid: Improving squeeze-and-excitation block with parametric sigmoid

Cited by 15 publications

References 38 publications

A split-federated learning and edge-cloud based efficient and privacy-preserving large-scale item recommendation model

A split-federated learning and edge-cloud based efficient and privacy-preserving large-scale item recommendation model

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

A Novel Deep Convolutional Neural Network Based on ResNet‐18 and Transfer Learning for Detection of Wood Knot Defects

Contact Info

Product

Resources

About