N-bit parity neural networks: new solutions based on linear programming

Liu, Derong; Hohil, Myron E.; Smith, Stanley H.

doi:10.1016/s0925-2312(01)00612-9

Cited by 22 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recurrent networks may solve some problems where there is a simple rule that generates the sequence of observations but general problems that are based on complex logic are not of this kind. Knowing beforehand that the data represents parity problem allows for setting an appropriate MLP or LSTM architecture to solve it [36,5,33,35,30,3,31,37,28,38], but for complex logic in real cases (for example, dynamics of brain processes) it will be very difficult to guess how to choose an appropriate model. Learning Boolean functions similar to parity may indeed be a great test for methods that try to evolve neural architecture to solve a given problem, but so far no such systems are in sight.…”

Section: Introductionmentioning

confidence: 99%

Separability is not the best goal for machine learning

Duch

2018

Preprint

View full text Add to dashboard Cite

Neural networks use their hidden layers to transform input data into linearly separable data clusters, with a linear or a perceptron type output layer making the final projection on the line perpendicular to the discriminating hyperplane. For complex data with multimodal distributions this transformation is difficult to learn. Projection on k ≥ 2 line segments is the simplest extension of linear separability, defining much easier goal for the learning process. Simple problems are 2-separable, but problems with inherent complex logic may be solved in a simple way by k-separable projections. The difficulty of learning nonlinear data distributions is shifted to separation of line intervals, simplifying the transformation of data by hidden network layers. For classification of difficult Boolean problems, such as the parity problem, linear projection combined with k-separability is sufficient and provides a powerful new target for learning. More complex targets may also be defined, changing the goal of learning from linear discrimination to creation of data distributions that can easily be handled by specialized models selected to analyze output distributions. This approach can replace many layers of transformation required by deep learning models.

show abstract

Section: Introductionmentioning

confidence: 99%

Separability is not the best goal for machine learning

Duch

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Otherwise it is the even parity problem. As the change of any bit of the vector causes different results consequently, the n-bit parity problem is very hard to solve, thus it is used widely for testing training algorithm of neural networks [3][4][5][6][7][8][9][10].…”

Section: Introductionmentioning

confidence: 99%

The upper bound of the minimal number of hidden neurons for the parity problem in binary neural networks

Yang

Wang

Huang

2011

Sci. China Inf. Sci.

View full text Add to dashboard Cite

Binary neural networks (BNNs) have important value in many application areas. They adopt linearly separable structures, which are simple and easy to implement by hardware. For a BNN with single hidden layer, the problem of how to determine the upper bound of the number of hidden neurons has not been solved well and truly. This paper defines a special structure called most isolated samples (MIS) in the Boolean space. We prove that at least 2 n−1 hidden neurons are needed to express the MIS logical relationship in the Boolean space if the hidden neurons of a BNN and its output neuron form a structure of AND/OR logic. Then the paper points out that the n-bit parity problem is just equivalent to the MIS structure. Furthermore, by proposing a new concept of restraining neuron and using it in the hidden layer, we can reduce the number of hidden neurons to n. This result explains the important role of restraining neurons in some cases. Finally, on the basis of Hamming sphere and SP function, both the restraining neuron and the n-bit parity problem are given a clear logical meaning, and can be described by a series of logical expressions.

show abstract

“…Special feedforward neural network architectures have been proposed to handle parity problems [16,30,31,28,21] but they are designed only for this special case and cannot be used for other Boolean functions, even very similar to parity. Learning systems are frequently tested on benchamrk datasets that are almost lienarly separable and relatively simple to handle, but without a strong prior knowledge it is very hard to find satisfactory solution for really complex problems.…”

Section: Introductionmentioning

confidence: 99%

Constructive Neural Network Algorithms That Solve Highly Non-separable Problems

Grochowski

Duch

2009

Constructive Neural Networks

View full text Add to dashboard Cite

Learning from data with complex non-local relations and multimodal class distribution for widely used classification algorithms is still very hard. Even if accurate solution is found the resulting model may be too complex for a given data and will not generalize well. New types of learning algorithms are needed to extend capabilities of standard machine learning systems. Projection pursuit methods can avoid "curse of dimensionality" by discovering interesting structures in lowdimensional subspace. This paper introduces constructive neural architectures based on projection pursuit techniques that are able to discover simplest models of data with inherent highly complex logical structures.

show abstract

N-bit parity neural networks: new solutions based on linear programming

Cited by 22 publications

References 9 publications

Separability is not the best goal for machine learning

Separability is not the best goal for machine learning

The upper bound of the minimal number of hidden neurons for the parity problem in binary neural networks

Constructive Neural Network Algorithms That Solve Highly Non-separable Problems

Contact Info

Product

Resources

About