Learning to overtake in TORCS using simple reinforcement learning

Loiacono, Daniele; Prete, Alessandro; Lanzi, Pier Luca; Cardamone, Luigi

doi:10.1109/cec.2010.5586191

Cited by 71 publications

(51 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Nonetheless, it is worth mentioning that in [10] a simple TDL method was implemented to learn only the overtaking behavior. In that system, all other tasks were adopted from an off-the-shelf controller.…”

Section: Introductionmentioning

confidence: 99%

Temporal difference learning with interpolated n-tuples: Initial results from a simulated car racing environment

Abdullahi

Lucas

2011

2011 IEEE Conference on Computational Intelligence and Games (CIG'11)

View full text Add to dashboard Cite

Evolutionary algorithms have been used successfully in car racing game competitions, such as the ones based on TORCS. This is in contrast to temporal difference learning (TDL), which despite being a powerful learning algorithm, has not been used to any significant extent within these competitions. We believe that this is mainly due to the difficulty of choosing a good function approximator, the potential instability of the learning behavior (and hence the reliability of the results), and the lack of a forward model which restricts the choice of TDL algorithms. This paper reports our initial results on using a new type of function approximator designed to be used with TDL for problems with a large number of continuous-valued inputs, where function approximators such as multi-layer perceptrons can be unstable. The approach combines interpolated tables with n-tuple systems. In order to conduct the research in a flexible and efficient way we developed a new car-racing simulator that runs much more quickly than TORCS and gives us full access to the forward model of the system.We investigate different types of tracks and physics models, and also make comparisons with human drivers and some initial tests with evolutionary learning (EL). The results show that each approach leads to different driving styles, and either TDL or EL can learn best depending on the details of the environment. Significantly, TDL produced best results when learning state-action values (similar to Q-learning; no forward model needed). Regarding driving style, TDL consistently learned behaviours that avoid damage while EL tended to evolve fast but reckless drivers.

show abstract

Section: Introductionmentioning

confidence: 99%

Temporal difference learning with interpolated n-tuples: Initial results from a simulated car racing environment

Abdullahi

Lucas

2011

2011 IEEE Conference on Computational Intelligence and Games (CIG'11)

View full text Add to dashboard Cite

show abstract

“…In Ref. 8, reinforcement learning techniques are used to achieve two complex racing behaviors: overtaking a fast opponent on a straight and overtaking on a tight bend.…”

Section: Introductionmentioning

confidence: 99%

An evolutionary tuned driving system for virtual car racing games: The AUTOPIA driver

Onieva¹,

Pelta²,

Godoy³

et al. 2012

Int. J. Intell. Syst.

View full text Add to dashboard Cite

This work presents a driving system designed for virtual racing situations. It is based on a complete modular architecture capable of automatically driving a car along a track with or without opponents. The architecture is composed of intuitive modules, with each one being responsible for a basic aspect of car driving. Moreover, this modularity of the architecture will allow us to replace or add modules in the future as a way to enhance particular features of particular situations. In the present work, some of the modules are implemented by means of hand-designed driving heuristics, whereas modules responsible for adapting the speed and direction of the vehicle to the track's shape, both critical aspects of driving a vehicle, are optimized by means of a genetic algorithm that evaluates the performance of the controller in four different tracks to obtain the best controller in a large number of situations; the algorithm also penalizes controllers that go out of the track, lose control, or get damaged. The evaluation of the performance is done in two ways. First, in runs with and without adversaries over several tracks. And second, the architecture was submitted as a participant to the 2010 Simulated Car Racing Competition, which in end won laurels. C 2012 Wiley Periodicals, Inc.

show abstract

“…More importantly, it also includes several pre-programmed bot drivers, and additional bots can be created essentially by defining a custom C function (Wymann, 2006). This makes it a convenient platform for research on autonomous vehicle control that has recently become quite popular (Munoz et al, 2009;Cardamone et al, 2009b;2009a;2010;Loiacono et al, 2010).…”

Section: Torcs Environmentmentioning

confidence: 99%

“…Of those, particularly relevant to this article are all studies on learning by imitation (Chambers and Michie, 1969;Urbancic and Bratko, 1994;Atkeson and Schaal, 1997;Bratko et al, 1998;D'Este et al, 2003;Sammut et al, 1992), especially those addressing the vehicle control task, either in the TORCS environment (Munoz et al, 2009;Cardamone et al 2009b;2009a;2010) or another simulated or real environment (Pomerleau, 1988;Togelius et al, 1996;Baluja, 1996). Other approaches to this task that do not follow the imitation learning scenario, including those based on reinforcement learning (Krödel and Kuhnert, 2002;Forbes, 2002;Loiacono et al, 2010), even if adopt substantially different assumptions about the available training information and use different learning algorithms, need to face the same crucial issues of state information and control action representation. In these respects, this work borrows substantially from many of those prior solutions.…”

Section: Related Workmentioning

confidence: 99%

“…This is because it can be used without precise and explicit domain knowledge and is capable of adapting to environment changes. While its potential utility has been demonstrated in a number of case studies (Pomerleau, 1988;Togelius et al, 1996;Baluja, 1996;Kröedel and Kuhnert, 2002;Forbes, 2002;Munoz et al, 2009;Cardamone et al, 2009b;2009a;2010;Loiacono et al, 2010), there is no single learning scenario and algorithm that proved universally successful and several possibilities remain unexplored. This article addresses the task of simulated racing car control and adopts the imitation learning scenario (Chambers and Michie, 1969;Urbancic and Bratko, 1994;Atkeson and Schaal, 1997;Bratko et al, 1998;D'Este et al, 2003;Sammut et al, 1992), in which an existing exemplary controller is used to generate training instances, needed to inductively learn a classification or regression model representing necessary control skills.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Imitation learning of car driving skills with decision trees and random forests

Cichosz

Pawełczak

2014

International Journal of Applied Mathematics and Computer Science

View full text Add to dashboard Cite

Machine learning is an appealing and useful approach to creating vehicle control algorithms, both for simulated and real vehicles. One common learning scenario that is often possible to apply is learning by imitation, in which the behavior of an exemplary driver provides training instances for a supervised learning algorithm. This article follows this approach in the domain of simulated car racing, using the TORCS simulator. In contrast to most prior work on imitation learning, a symbolic decision tree knowledge representation is adopted, which combines potentially high accuracy with human readability, an advantage that can be important in many applications. Decision trees are demonstrated to be capable of representing high quality control models, reaching the performance level of sophisticated pre-designed algorithms. This is achieved by enhancing the basic imitation learning scenario to include active retraining, automatically triggered on control failures. It is also demonstrated how better stability and generalization can be achieved by sacrificing human-readability and using decision tree model ensembles. The methodology for learning control models contributed by this article can be hopefully applied to solve real-world control tasks, as well as to develop video game bots.

show abstract

Learning to overtake in TORCS using simple reinforcement learning

Cited by 71 publications

References 17 publications

Temporal difference learning with interpolated n-tuples: Initial results from a simulated car racing environment

Temporal difference learning with interpolated n-tuples: Initial results from a simulated car racing environment

An evolutionary tuned driving system for virtual car racing games: The AUTOPIA driver

Imitation learning of car driving skills with decision trees and random forests

Contact Info

Product

Resources

About