Representation Learning for Context-Dependent Decision-Making

Qin, Yuzhen; Menara, Tommaso; Oymak, Samet; Ching, ShiNung; Pasqualetti, Fabio

doi:10.23919/acc53348.2022.9867204

Cited by 3 publications

(1 citation statement)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The piece-wise stationary stochastic multi-armed bandits have also been considered for the linear bandits; the linear bandit is when the reward is the inner product of an unknown vector and an action vector chosen by the learner [8]. Prior work has considered the cases where the unknown parameter vector changes abruptly [9], and where the rewards are dependent on the previously chosen action vectors [10]. Some non-stationary methods focused on the impact of distributional change in the form of variational budget, which is a known constant that bounds the cumulative changes in the distributions [11], [12].…”

Section: Introductionmentioning

confidence: 99%

Stochastic Multi-Armed Bandits with Non-Stationary Rewards Generated by a Linear Dynamical System

Jonathan

Hosseinzadeh

Sinopoli

2022

2022 IEEE 61st Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

Online decision-making can be formulated as the popular stochastic multi-armed bandit problem where a learner makes decisions (or takes actions) to maximize cumulative rewards collected from an unknown environment. A specific variant of the bandit problem is the non-stationary stochastic multiarmed bandit problem, where the reward distributions-which are unknown to the learner-change over time. This paper proposes to model non-stationary stochastic multi-armed bandits as an unknown stochastic linear dynamical system as many applications, such as bandits for dynamic pricing problems or hyperparameter selection for machine learning models, can benefit from this perspective. Following this approach, we can build a matrix representation of the system's steady-state Kalman filter that takes a set of previously collected observations from a time interval of length s to predict the next reward that will be returned for each action. This paper proposes a solution in which the parameter s is determined via an adaptive algorithm by analyzing the model uncertainty of the matrix representation. This algorithm helps the learner adaptively adjust its model size and its length of exploration based on the uncertainty of its environmental model. The effectiveness of the proposed scheme is demonstrated through extensive numerical studies, revealing that the proposed scheme is capable of increasing the rate of collected cumulative rewards.

show abstract

Section: Introductionmentioning

confidence: 99%

Stochastic Multi-Armed Bandits with Non-Stationary Rewards Generated by a Linear Dynamical System

Jonathan

Hosseinzadeh

Sinopoli

2022

2022 IEEE 61st Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

show abstract

Non-Stationary Representation Learning in Sequential Linear Bandits

Qin

Menara

Oymak

et al. 2022

IEEE Open J. Control. Syst.

View full text Add to dashboard Cite

In this paper, we study representation learning for multi-task decision-making in non-stationary environments. We consider the framework of sequential linear bandits, where the agent performs a series of tasks drawn from different environments. The embeddings of tasks in each environment share a lowdimensional feature extractor called representation, and representations are different across environments. We propose an online algorithm that facilitates efficient decision-making by learning and transferring nonstationary representations in an adaptive fashion. We prove that our algorithm significantly outperforms the existing ones that treat tasks independently. We also conduct experiments using both synthetic and real data to validate our theoretical insights and demonstrate the efficacy of our algorithm.

show abstract

Control of Linear-Threshold Brain Networks via Reservoir Computing

McCreesh,

Cortés

2024

IEEE Open J. Control. Syst.

View full text Add to dashboard Cite

Representation Learning for Context-Dependent Decision-Making

Cited by 3 publications

References 15 publications

Stochastic Multi-Armed Bandits with Non-Stationary Rewards Generated by a Linear Dynamical System

Stochastic Multi-Armed Bandits with Non-Stationary Rewards Generated by a Linear Dynamical System

Non-Stationary Representation Learning in Sequential Linear Bandits

Control of Linear-Threshold Brain Networks via Reservoir Computing

Contact Info

Product

Resources

About