Leveraging Human Guidance for Deep Reinforcement Learning Tasks

Zhang, Ruohan; Torabi, Faraz; Guan, Lin; Ballard, Dana H.; Stone, Peter

doi:10.24963/ijcai.2019/884

Cited by 45 publications

(26 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The policy can be learned through trial and error (RL) or from an expert's demonstration (IL). A major issue of RL is its sample inefficiency and human demonstration has been shown to speed up learning (Silver et al 2016;Hester et al 2018;de la Cruz, Du, and Taylor 2018;Zhang et al 2019).…”

Section: Introductionmentioning

confidence: 99%

Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset

Zhang

Walshe

Liu

et al. 2020

AAAI

Self Cite

View full text Add to dashboard Cite

Large-scale public datasets have been shown to benefit research in multiple areas of modern artificial intelligence. For decision-making research that requires human data, high-quality datasets serve as important benchmarks to facilitate the development of new methods by providing a common reproducible standard. Many human decision-making tasks require visual attention to obtain high levels of performance. Therefore, measuring eye movements can provide a rich source of information about the strategies that humans use to solve decision-making tasks. Here, we provide a large-scale, high-quality dataset of human actions with simultaneously recorded eye movements while humans play Atari video games. The dataset consists of 117 hours of gameplay data from a diverse set of 20 games, with 8 million action demonstrations and 328 million gaze samples. We introduce a novel form of gameplay, in which the human plays in a semi-frame-by-frame manner. This leads to near-optimal game decisions and game scores that are comparable or better than known human records. We demonstrate the usefulness of the dataset through two simple applications: predicting human gaze and imitating human demonstrated actions. The quality of the data leads to promising results in both tasks. Moreover, using a learned human gaze model to inform imitation learning leads to an 115% increase in game performance. We interpret these results as highlighting the importance of incorporating human visual attention in models of decision making and demonstrating the value of the current dataset to the research community. We hope that the scale and quality of this dataset can provide more opportunities to researchers in the areas of visual attention, imitation learning, and reinforcement learning.

show abstract

Section: Introductionmentioning

confidence: 99%

Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset

Zhang

Walshe

Liu

et al. 2020

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…IRL has become an important apprenticeship approach to speed up convergence in classic RL problems utilizing expert and non-expert knowledge, experience and preferences [25]. In this context, four types of human guidance have been identified [27]: (i) standard imitation learning, in which the human trainer observes state information and demonstrates action to the agent, while the agent stores this data to be used in learning later; (ii) learning from evaluative feedback, in which the human trainer watches the agent performing the task, and provides instant feedback on agent decision in the associated state; (iii) imitation from observation, in which, in contrast to standard imitation, the agent does not have access to human demonstrated action; and, (iv) learning attention from human, where the trainer provides attention map to the learning agent.…”

Section: B Interactive Reinforcement Learningmentioning

confidence: 99%

“…One of the main challenges in this approach is to interpret human feedback correctly since such interpretation determines how the feedback is used to improve the policy in the MDP framework [25]. The main categories of methods used for incorporating evaluative human feedback in IRL are [27]: Policy Shaping, Reward Shaping, Intervention, and Policy-dependent Feedback.…”

Section: B Interactive Reinforcement Learningmentioning

confidence: 99%

Human-Augmented Prescriptive Analytics With Interactive Multi-Objective Reinforcement Learning

et al. 2021

View full text Add to dashboard Cite

The rise of Artificial Intelligence (AI) enables enterprises to manage large amounts of data in order to derive predictions about future performance and to gain meaningful insights. In this context, descriptive and predictive analytics has gained a significant research attention; however, prescriptive analytics has just started to emerge as the next step towards increasing data analytics maturity and leading to optimized decision making ahead of time. Although machine learning for decision making has been identified as one of the most important applications of AI, up to now, prescriptive analytics is mainly addressed with domain-specific optimization models. On the other hand, existing literature lacks generalized prescriptive analytics models capable of being dynamically adapted according to the human preferences. Reinforcement Learning, as the third machine learning paradigm alongside supervised learning and unsupervised learning, has the potential to deal with the dynamic, uncertain and time-variant environments, the huge states space of sequential decision making processes, as well as the incomplete knowledge. In this paper, we propose a human-augmented prescriptive analytics approach using Interactive Multi-Objective Reinforcement Learning (IMORL) in order to cope with the complexity of real-life environments and the need for optimized human-machine collaborations. The decision making process is modelled in a generalized way in order to assure scalability and applicability in a wide range of problems and applications. We deployed the proposed approach in a stock market case study in order to evaluate the proactive trading decisions that will lead to the maximum return and the minimum risk that the user's experience and the available data can yield in combination.INDEX TERMS Big data, data analytics, human-augmented analytics, machine learning, prescriptive analytics, reinforcement learning.

show abstract

“…2: The increasing research interest in the human-inthe-loop, obtained through Google scholar search with keywords: "human-in-the-loop" and "machine learning". such as clinical diagnosis and lack of training data [33,34,35,36].…”

Section: Introductionmentioning

confidence: 99%

A Survey of Human-in-the-loop for Machine Learning

Wu,

Xiao,

Sun

et al. 2021

Preprint

View full text Add to dashboard Cite

Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish some tasks that are hard for computers in the pipeline with the help of machine-based approaches. In this paper, we survey existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. Using the above categorization, we summarize major approaches in the field, along with their technical strengths/ weaknesses, we have simple classification and discussion in natural language processing, computer vision, and others. Besides, we provide some open challenges and opportunities. This survey intends to provide a high-level summarization for human-in-the-loop and motivates interested readers to consider approaches for designing effective human-in-the-loop solutions.

show abstract

Leveraging Human Guidance for Deep Reinforcement Learning Tasks

Cited by 45 publications

References 57 publications

Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset

Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset

Human-Augmented Prescriptive Analytics With Interactive Multi-Objective Reinforcement Learning

A Survey of Human-in-the-loop for Machine Learning

Contact Info

Product

Resources

About