On practical constraints of approximation using neural networks on current digital computers

Puheim, Michal; Nyulaszi, Ladislav; Madarasz, L.; Gaspar, V.

doi:10.1109/ines.2014.6909379

Cited by 4 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of input neurons in each MLP will be the same as a number of preceding concepts and there will be single output, as there is only one single concept which is being influenced. A topology of the deployed MLPs is to be considered, but generally even small and simple 2-hidden-layer topologies, with less than 5 neurons in each hidden layer, should perform better than the conventional linear FCM relations as their approximation capability is greater [13]. It is also possible to use MLPs with no hidden layer.…”

Section: A Nonlinear Relations Represented By Mlpsmentioning

confidence: 99%

“…The replacement of the linear relations with the MLPs naturally leads to an increased computational complexity of the resulting FCM. This is obvious, as the increased approximation capability cannot be achieved without an increase in the model complexity [13]. The increase in the complexity is equal to the sum of the computational complexities of all used MLPs and grows exponentially with the size of the used MLPs.…”

Section: A Nonlinear Relations Represented By Mlpsmentioning

confidence: 99%

See 1 more Smart Citation

Three-term relation neuro-fuzzy cognitive maps

Puheim

Vaščák

Madarasz

2014

2014 IEEE 15th International Symposium on Computational Intelligence and Informatics (CINTI)

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a novel approach to modeling using fuzzy cognitive maps, which we refer to as the Three-Term Relation Neuro-Fuzzy Cognitive Map or simply the TTR NFCM. The proposed method is mostly suited to model complex nonlinear technical systems with dynamic internal characteristics. With this method we aim to solve some of the most critical problems of the conventional fuzzy cognitive maps. We target two of these problems by hybridization with artificial neural networks. First of them is a linear nature of relations between the concepts. The second is a lack of mutual dependence between the relations connecting to the same concept. Finally, we tackle a problem of relation dynamics using an inspiration from the control engineering. While focusing on bringing these advanced additional methods to the design of cognitive maps, we also aim to keep the degree of dependency on expert knowledge on the same level as with the conventional fuzzy cognitive maps. We achieve this by utilizing the machine learning methods. However, since the proposed method is heavily dependent on automated data-driven learning, it is suitable mainly for systems which are well observable and can produce sufficient training datasets.

show abstract

Section: A Nonlinear Relations Represented By Mlpsmentioning

confidence: 99%

Section: A Nonlinear Relations Represented By Mlpsmentioning

confidence: 99%

Three-term relation neuro-fuzzy cognitive maps

Puheim

Vaščák

Madarasz

2014

2014 IEEE 15th International Symposium on Computational Intelligence and Informatics (CINTI)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Therefore, φ θ1,1,z (x; F p,q ) = φ θ1,2,z (x; F p,q ) = 0, (20) which leads to f 1 (x; F p,q ) = 0. If 2 −p+emin ≤ x ≤ Ω, we have…”

Section: Representability Test For Case 1-3 (1)mentioning

confidence: 99%

“…Another line of work on memory capacity has studied regression problems and showed that shallow networks with O(n) parameters can fit arbitrary n input/output pairs [4,11,23,26], while o(n) parameters are sufficient for deep ones, i.e., ω(1) layers [17,22]. However, since these results assume exact mathematical operations, they do not apply to neural networks executed on computers that can only represent a tiny subset of the reals (e.g., floating-point numbers) and perform inexact operations (e.g., floating-point operations) [24,20].…”

Section: Introductionmentioning

confidence: 99%

The Effect of Floating Wetland on Water Quality Improvement in a Eutrophic Lake.

Park¹,

Park²,

Choi³

et al. 2013

KJEE

View full text Add to dashboard Cite

The study of the expressive power of neural networks has investigated the fundamental limits of neural networks. Most existing results assume real-valued inputs and parameters as well as exact operations during the evaluation of neural networks. However, neural networks are typically executed on computers that can only represent a tiny subset of the reals and apply inexact operations. In this work, we analyze the expressive power of neural networks under a more realistic setup: when we use floating-point numbers and operations. Our first set of results assumes floating-point operations where the significand of a float is represented by finite bits but its exponent can take any integer value. Under this setup, we show that neural networks using a binary threshold unit or ReLU can memorize any finite input/output pairs and can approximate any continuous function within a small error. We also show similar results on memorization and universal approximation when floating-point operations use finite bits for both significand and exponent; these results are applicable to many popular floating-point formats such as those defined in the IEEE 754 standard (e.g., 32-bit single-precision format) and bfloat16.

show abstract