Associative learning is a fundamental cognitive capacity that allows animals and humans to learn the predictive relationship between behavioral events and rewarding outcomes. While the process of learning is commonly conceptualized as a prospective strategy (learning how behavioral events predict future rewards), here we provide behavioral and neurophysiological evidence to show that animals may instead employ a retrospective and stepwise learning strategy (learning how the reward is predicted by preceding behavioral events). In rats learning a new association in which the reward was paired with a sequence of behavioral events, learning started from the event closest to the reward and sequentially incorporated earlier events into animals' internal model. The learning of each behavioral event as a new reward predictor was accompanied by the emergence of basal forebrain (BF) neuronal responses toward that event. BF activities quantitatively conveyed a reward prediction error signal associated with the behavioral event, and promoted reward-seeking behavioral sequences containing the newly learned event. As the internal model incorporated more behavioral events as reward predictors, non-rewarded behavioral sequences that were once compatible with the internal model during early stages of learning became incompatible and were sequentially eliminated. Together, these results demonstrate how the retrospective and stepwise learning strategy can effectively establish animals' internal model during the learning process and lead to the sequential refinement of reward-seeking behaviors. These results also highlight the functional significance of BF neuronal activities, which provided unique insights into the covert dynamics of the learning process in single trials.