Abstract-Sequential learning problems such as speech, cursive handwriting, time series forecasting and protein sequence prediction. Both Speech and cursive handwriting recognition are challenging problems to Pattern recognition systems, in particular speech signal. Some peculiar characteristics of these types of problems are that, the signal or pattern evolves with time, modeling a long time dependencies in this pattern is a major challenge. Hidden Markov models (HMM) have been applied for these types of problems. Due to some obvious shortcomings of HMM, neural networks were also explored and applied as well as their hybrids. The problem of feature variability in sequence learning is still a challenging problem. In this paper, we analyzed the problem, present some methods in feature variance suppression in character recognition, and review some research efforts in modification of neural networks and applications. We proposed a structure for a state-based neural network.Index Terms-Sequence learning, feature variability, neural network.