Learning from hints in neural networks

Abstract. Financial forecasting is an example of a signal processing problem which is challenging due to small sample sizes, high noise, non-stationarity, and non-linearity. Neural networks have been very successful in a number of signal processing applications. We discuss fundamental limitations and inherent difficulties when using neural networks for the processing of high noise, small sample size signals. We introduce a new intelligent signal processing method which addresses the difficulties. The method proposed uses conversion into a symbolic representation with a self-organizing map, and grammatical inference with recurrent neural networks. We apply the method to the prediction of daily foreign exchange rates, addressing difficulties with non-stationarity, overfitting, and unequal a priori class probabilities, and we find significant predictability in comprehensive experiments covering 5 different foreign exchange rates. The method correctly predicts the direction of change for the next day with an error rate of 47.1%. The error rate reduces to around 40% when rejecting examples where the system has low confidence in its prediction. We show that the symbolic representation aids the extraction of symbolic knowledge from the trained recurrent neural networks in the form of deterministic finite state automata. These automata explain the operation of the system and are often relatively simple. Automata rules related to well known behavior such as trend following and mean reversal are extracted.

show abstract

“…This can be seen as similar to the use of hints (Abu-Mostafa, 1990) and should help to make the problem less ill-posed.…”

Section: Fundamental Issuesmentioning

confidence: 97%

Untitled

2001

View full text Add to dashboard Cite

show abstract

“…In image, text analysis or robotics many methods have been devised for knowledge transfer. Related machine learning subjects include: learning from hints [104], lifelong learning [105], multi-task learning [106], cross-domain learning [107,108], cross-category learning [109] and selftaught learning [110]. EigenTransfer algorithm [111] tries to unify various transfer learning ideas representing the target task by a graph.…”

Section: Transfer Of Knowledgementioning

confidence: 99%

Optimal Support Features for Meta-Learning

Duch

Maszczyk

Grochowski

2011

Studies in Computational Intelligence

View full text Add to dashboard Cite

Meta-learning has many aspects, but its final goal is to discover in an automatic way many interesting models for a given data. Our early attempts in this area involved heterogeneous learning systems combined with a complexity-guided search for optimal models, performed within the framework of (dis)similarity based methods to discover "knowledge granules". This approach, inspired by neurocognitive mechanisms of information processing in the brain, is generalized here to learning based on parallel chains of transformations that extract useful information granules and use it as additional features. Various types of transformations that generate hidden features are analyzed and methods to generate them are discussed. They include restricted random projections, optimization of these features using projection pursuit methods, similarity-based and general kernel-based features, conditionally defined features, features derived from partial successes of various learning algorithms, and using the whole learning models as new features. In the enhanced feature space the goal of learning is to create image of the input data that can be directly handled by relatively simple decision processes. The focus is on hierarchical methods for generation of information, starting from new support features that are discovered by different types of data models created on similar tasks and successively building more complex features on the enhanced feature spaces. Resulting algorithms facilitate deep learning, and also enable understanding of structures present in the data by visualization of the results of data transformations and by creating logical, fuzzy and prototype-based rules based on new features. Relations to various machine-learning approaches, comparison of results, and neurocognitive inspirations for meta-learning are discussed.

show abstract

“…Learning from partial information is referred to as "learning from hints". Although the availability of that information does not overcome the theoretical limitations arising from computational complexity arguments (Abu-Mostafa, 1990), there is no doubt about the actual role of learning from hints in practice. Interesting "hints" for solving special problems have been proposed in (A1- Mashouq & Reed, 1991;Omlin & Giles, 1992).…”

Section: Finite Automata and Recurrent Networkmentioning

confidence: 99%

Representation of finite state automata in Recurrent Radial Basis Function networks

1996

View full text Add to dashboard Cite

Abstract. In this paper, we propose some techniques for injecting finite state automata into Recurrent Radial Basis Function networks (R2BF). When providing proper hints and constraining the weight space properly, we show that these networks behave as automata. A technique is suggested for forcing the learning process to develop automata representations that is based on adding a proper penalty function to the ordinary cost. Successful experimental results are shown for inductive inference of regular grammars.

show abstract

Learning from hints in neural networks

Cited by 186 publications

References 6 publications

Untitled

Untitled

Optimal Support Features for Meta-Learning

Representation of finite state automata in Recurrent Radial Basis Function networks

Contact Info

Product

Resources

About