Time-aware deep reinforcement learning with multi-temporal abstraction

Abstract

KIM, YEO JIN. Time-aware Deep Reinforcement Learning with Multi-Temporal Abstraction. (Under the direction of Dr. Min Chi).Reinforcement learning (RL) is an effective learning mechanism for handling sequential decisionmaking tasks. Specifically, deep reinforcement learning (DRL) has achieved great success in diverse complex tasks such as games and robotics. While a conventional DRL framework premises 1) complete data, 2) regular time intervals, and 3) full observations to induce an optimal policy, such premises often do not hold in many real-world tasks with offline irregular time series. In this dissertation, we tackled the three major challenges: 1) missing data, 2) temporal irregularity, and 3) partial observations in offline sequential data.First, heterogeneous features having different observation frequencies cause a high rate of missing data, which deteriorates the data quality and increases the uncertainty of decisions. To impute missing values, we propose Temporal Belief Memory (TBM), which is a bio-inspired missing data imputation method based on the temporal belief of observations. Our results show that TBM outperforms Missing Indicators, the state-of-the-art missing data handling method, on our septic shock prediction task.Second, real-world sequential data are often collected at irregular time intervals. Without consideration of continuous time intervals on discrete time steps, the agent is deficient to discern different temporal contexts and often fails to accurately estimate states and rewards. To address this issue, we propose a Time-aware DRL framework, which estimates states and rewards by explicitly representing time intervals between consecutive observations in the learning framework. We evaluated the proposed framework on two classic RL problems, Atari games, and two real-world tasks including nuclear reactor operation and septic shock prevention, and the results show the Time-aware DRL framework significantly improves performance in all the tasks. Third, in partially observable environments, a single view of temporal sequences generally limits observation experience and is biased to a specific series of observations, while a multi-view learns a more comprehensive representation than a single view and thus improves generalization performance. Also, in long-horizon decision making problems, temporal abstraction enables the agent to effectively learn long-range dependency by skipping over the high-frequency details. To take each advantage of multi-view and temporal abstraction, we propose a Multi-Temporal Abstraction (MTA) mechanism, which provides diverse temporal views over various periods of dependencies for state approximation. MTA provides three temporal modes: Retrospective, Concurrent, and Proactive modes, conditional on the temporal properties of target problems. Our results demonstrate that the Time-aware DRL framework with MTA (T-MTA) significantly increases training efficiency and effectiveness on two aforementioned real-world tasks.