Why Squashing Functions in Multi-Layer Neural Networks

Urenda, Julio C.; Csiszár, Orsolya; Csiszár, Gábor; Dombi, József; Kosheleva, Olga; Kreinovich, Vladik; Eigner, György

doi:10.1109/smc42975.2020.9282886

Cited by 7 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This model also seems to explain the great success of the rectified linear unit (ReLU). In [33], the authors elaborate on the empirical success of squashing functions in neural networks by showing that the formulae describing this family follow from natural symmetry requirements.…”

Section: Neural Network Based On Nilpotent Fuzzy Logic and Mcdm Toolsmentioning

confidence: 99%

Uninorm-like parametric activation functions for human-understandable neural models

Csiszár¹,

Sára²,

Dénes-Fazakas³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We present a deep learning model for finding human-understandable connections between input features. Our approach uses a parameterized, differentiable activation function, based on the theoretical background of nilpotent fuzzy logic and multi-criteria decision-making (MCDM). The learnable parameter has a semantic meaning indicating the level of compensation between input features. The neural network determines the parameters using gradient descent to find human-understandable relationships between input features. We demonstrate the utility and effectiveness of the model by successfully applying it to classification problems from the UCI Machine Learning Repository.

show abstract

Section: Neural Network Based On Nilpotent Fuzzy Logic and Mcdm Toolsmentioning

confidence: 99%

Uninorm-like parametric activation functions for human-understandable neural models

Csiszár¹,

Sára²,

Dénes-Fazakas³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Observe that in this model we have implemented rectified linear layers. Recent investigations have demonstrated that rectified linear functions are the most effective in representing data processing in neural networks 5 , particularly for networks with many layers (Urenda et al, 2020). Additionally, the ordinal classification problem of the therapy length is, based on Kramer et al, handled as a regression problem with an additional postprocessing step (Kramer et al, 2001).…”

Section: Baseline Modelsmentioning

confidence: 99%

Bayesian Logical Neural Networks for Human-Centered Applications in Medicine

Ochoa

Майер

Csiszar

2021

Preprint

View full text Add to dashboard Cite

Medicine is characterized by its inherent ambiguity, i.e., the difficulty to identify and obtain exact outcomes from available data. Regarding this problem, electronic Health Records (EHRs) aim to avoid imprecisions in the data recording, for instance by its recording in an automatic way or by the integration of data that is both readable by humans and machines. However, the inherent biology and physiological processes introduce a constant epistemic uncertainty, which has a deep implication in the way the condition of the patients is estimated. For instance, for some patients, it is not possible to speak about an exact diagnosis, but about the “suspicion” of a disease, which reveals that the medical practice is often ambiguous. In this work, we report a novel modeling methodology combining explainable models, defined on Logic Neural Networks (LONNs), and Bayesian Networks (BN) that deliver ambiguous outcomes, for instance, medical procedures (Therapy Keys (TK)), depending on the uncertainty of observed data. If epistemic uncertainty is generated from the underlying physiology, the model delivers exact or ambiguous results depending on the individual parameters of each patient. Thus, our model does not aim to assist the customer by providing exact results but is a user-centered solution that informs the customer when a given recommendation, in this case, a therapy, is uncertain and must be carefully evaluated by the customer, implying that the final customer must be a professional who will not fully rely on automatic recommendations. This novel methodology has been tested on a database for patients with heart insufficiency.

show abstract

“…In mathematical terms, this means that the corresponding group must be finite-dimensional. It is known that under reasonable conditions, any finite-dimensional transformation group that contains all linear transformation contains only fractional-linear transformations, i.e., transformations of the type [6,7,10,11], etc.…”

Section: What Probability Distributions Satisfy This Invariance Requirement?mentioning

confidence: 99%

Why Kappa Regression?

Urenda¹,

Csiszár²,

Csiszár³

et al. 2021

Atlantis Studies in Uncertainty Modelling

Self Cite

View full text Add to dashboard Cite

A recent book provides examples that a new class of probability distributions and membership functions -called kappa-regression distributions and membership functionsleads, in many practical applications, to better data processing results than using previously known classes. In this paper, we provide a theoretical explanation for this empirical success -namely, we show that these distributions are the only ones that satisfy reasonable invariance requirements.

show abstract

Why Squashing Functions in Multi-Layer Neural Networks

Cited by 7 publications

References 8 publications

Uninorm-like parametric activation functions for human-understandable neural models

Uninorm-like parametric activation functions for human-understandable neural models

Bayesian Logical Neural Networks for Human-Centered Applications in Medicine

Why Kappa Regression?

Contact Info

Product

Resources

About