A simple and efficient architecture for trainable activation functions

Apicella, Andrea; Isgrò, Francesco; Prevete, Roberto

doi:10.1016/j.neucom.2019.08.065

Cited by 36 publications

(13 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Work towards the enhancement of activation functions in the neural networks was also proposed, such as Variable Activation Function (VAF) [17] and Adaptive Takagi-Sugeno-Kang (AdaTSK) [23]. Apart from those adaptive action functions, [15] presented a proposition of two-layer mixture of factor analysers with joint factor loading (2L-MJFA) for conducting the dimensionality reduction and classification together.…”

Section: Machine Learning For Digital Healthcarementioning

confidence: 99%

See 1 more Smart Citation

Curvature-based Feature Selection with Application in Classifying Electronic Health Records

Zuo,

Li,

et al. 2021

Preprint

View full text Add to dashboard Cite

Electronic Health Records (EHRs) are widely applied in healthcare facilities nowadays. Due to the inherent heterogeneity, unbalanced, incompleteness, and high-dimensional nature of EHRs, it is a challenging task to employ machine learning algorithms to analyse such EHRs for prediction and diagnostics within the scope of precision medicine. Dimensionality reduction is an efficient data preprocessing technique for the analysis of high dimensional data that reduces the number of features while improving the performance of the data analysis, e.g. classification. In this paper, we propose an efficient curvature-based feature selection method for supporting more precise diagnosis. The proposed method is a filter-based feature selection method, which directly utilises the Menger Curvature for ranking all the attributes in the given data set. We evaluate the performance of our method against conventional PCA and recent ones including BPCM, GSAM, WCNN, BLS II, VIBES, 2L-MJFA, RFGA, and VAF. Our method achieves state-of-the-art performance on four benchmark healthcare data sets including CCRFDS, BCCDS, BTDS, and DRDDS with impressive 24.73% and 13.93% improvements respectively on BTDS and CCRFDS, 7.97% improvement on BCCDS, and 3.63% improvement on DRDDS. Our CFS source code is publicly available at https://github.com/zhemingzuo/CFS. Keywords feature selection • precision medicine • healthcare • electronic health records • classification * Equal contribution.

show abstract

Section: Machine Learning For Digital Healthcarementioning

confidence: 99%

“…Recently, pervasive healthcare becomes the central topic which attracts intensive attentions and interests from academia, industry, as well as healthcare sectors [10,11,12,13,14,15,16,17]. In this problem domain, highly class-imbalanced data set with a large number of missing values are common problems [18].…”

Section: Introductionmentioning

confidence: 99%

Curvature-based Feature Selection with Application in Classifying Electronic Health Records

Zuo,

Li,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Variable Activation Function. In (Apicella et al, 2019) trainable activation functions are expressed in terms of sub-networks with only one hidden layer, relying on the consideration that a one-hidden layer neural network can approximate arbitrarily well any continuous functional mapping from one finite-dimensional space to another, enabling the resulting function to assume "any" shape. In a nutshell, the proposed activation is modelled as a non-linear activation function f with a neuron with an Identity activation function which sends its output to a one-hidden-layer sub-network with just one output neuron having, in turn, an Identity as an output function.…”

Section: Linear Combination Of One-to-one Functionsmentioning

confidence: 99%

“…In other words, the key idea is to involve the activation functions in the learning process together (or separately) with the other parameters of the network such as weights and biases, thus obtaining a trained activation function. In the literature we usually find the expression "trainable activation functions", however the expressions "learneable", "adaptive" or "adaptable" activation functions are also used, see, for example, (Scardapane et al, 2018;Apicella et al, 2019;Qian et al, 2018). Many and heterogeneous trainable activation function models have been proposed in the literature, and in recent years there is a particular interest in this topic, see Figure 1.…”

Section: Introductionmentioning

confidence: 99%

A survey on modern trainable activation functions

Apicella,

Donnarumma,

Isgrò

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

In the literature, there is a strong interest to identify and define activation functions which can improve neural network performance. In recent years there is a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred as trainable, learnable or adaptable activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss on main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to add neuron layers which use fixed activation functions (nontrainable activation functions) and some simple local rule constrains the corresponding weight layers.

show abstract

“…Even though transfer learning (i.e., the use of pre-trained CNNs on large-scale datasets of natural images) could be applied, hundreds of accurately annotated input samples should be available [28]. Therefore, parameter-efficient architectures, including simple trainable activation functions [29] or mixed-scale dense CNNs [30], might be beneficial to deal with the paucity of manually labeled and validated datasets. Alternatively, also data augmentation techniques based on Generative Adversarial Networks (GANs) [31,32] or interactive solutions [33], require time-consuming annotation by experts.…”

Section: Introductionmentioning

confidence: 99%

ACDC: Automated Cell Detection and Counting for Time-Lapse Fluorescence Microscopy

et al. 2020

View full text Add to dashboard Cite

Advances in microscopy imaging technologies have enabled the visualization of live-cell dynamic processes using time-lapse microscopy imaging. However, modern methods exhibit several limitations related to the training phases and to time constraints, hindering their application in the laboratory practice. In this work, we present a novel method, named Automated Cell Detection and Counting (ACDC), designed for activity detection of fluorescent labeled cell nuclei in time-lapse microscopy. ACDC overcomes the limitations of the literature methods, by first applying bilateral filtering on the original image to smooth the input cell images while preserving edge sharpness, and then by exploiting the watershed transform and morphological filtering. Moreover, ACDC represents a feasible solution for the laboratory practice, as it can leverage multi-core architectures in computer clusters to efficiently handle large-scale imaging datasets. Indeed, our Parent-Workers implementation of ACDC allows to obtain up to a 3.7× speed-up compared to the sequential counterpart. ACDC was tested on two distinct cell imaging datasets to assess its accuracy and effectiveness on images with different characteristics. We achieved an accurate cell-count and nuclei segmentation without relying on large-scale annotated datasets, a result confirmed by the average Dice Similarity Coefficients of 76.84 and 88.64 and the Pearson coefficients of 0.99 and 0.96, calculated against the manual cell counting, on the two tested datasets.

show abstract

A simple and efficient architecture for trainable activation functions

Cited by 36 publications

References 30 publications

Curvature-based Feature Selection with Application in Classifying Electronic Health Records

Curvature-based Feature Selection with Application in Classifying Electronic Health Records

A survey on modern trainable activation functions

ACDC: Automated Cell Detection and Counting for Time-Lapse Fluorescence Microscopy

Contact Info

Product

Resources

About