Incorporating conflicting descriptions into decisions from experience

Weiss‐Cohen, Leonardo; Konstantinidis, Emmanouil; Speekenbrink, Maarten; Harvey, Nigel

doi:10.1016/j.obhdp.2016.05.005

“…For example, recent studies in experiencebased decision-making provided participants with descriptions about the underlying distributions that generate rewards (e.g., Lejarraga & Gonzalez, 2011;Weiss-Cohen, Konstantinidis, Speekenbrink, & Harvey, 2016). Just as in the CMAB, this presents a naturalistic decision environment in which different sources of information (e.g., descriptions and participants' own experience) need to be integrated in order to choose between alternatives or courses of action.…”

Section: Contextual Multi-armed Banditsmentioning

confidence: 99%

Putting bandits into context: How function learning supports decision making.

Schulz¹,

Konstantinidis

²

,

Speekenbrink³

2018

Journal of Experimental Psychology: Learning, Memory, and Cogni

Self Cite

View full text Add to dashboard Cite

We introduce the contextual multi-armed bandit task as a framework to investigate learning and decision making in uncertain environments. In this novel paradigm, participants repeatedly choose between multiple options in order to maximise their rewards. The options are described by a number of contextual features which are predictive of the rewards through initially unknown functions. From their experience with choosing options and observing the consequences of their decisions, participants can learn about the functional relation between contexts and rewards and improve their decision strategy over time. In three experiments, we explore participants' behaviour in such learning environments. We predict participants' behaviour by context-blind (mean-tracking, Kalman filter) and contextual (Gaussian process and linear regression) learning approaches combined with different choice strategies. Participants are mostly able to learn about the context-reward functions and their behaviour is best described by a Gaussian process learning strategy which generalizes previous experience to similar instances. In a relatively simple task with binary features, they seem to combine this learning with a "probability of improvement" decision strategy which focuses on alternatives that are expected to lead to an improvement upon a current favourite option. In a task with continuous features that are linearly related to the rewards, participants seem to more explicitly balance exploration and exploitation. Finally, in a difficult learning environment where the relation between features and rewards is non-linear, some participants are again well-described by a Gaussian process learning strategy, whereas others revert to context-blind strategies.

show abstract

“…While the CMAB is novel in the psychological literature (though see Schulz, Konstantinidis, & Speekenbrink, 2015;Stojic, Analytis, & Speekenbrink, 2015), where few tasks explicitly combine function learning and experience-based decision making, there are certain similarities with tasks used in previous research. For example, recent studies in experiencebased decision-making provided participants with descriptions about the underlying distributions that generate rewards (e.g., Lejarraga & Gonzalez, 2011;Weiss-Cohen, Konstantinidis, Speekenbrink, & Harvey, 2016). Just as in the CMAB, this presents a naturalistic decision environment in which different sources of information (e.g., descriptions and participants' own experience) need to be integrated in order to choose between alternatives or courses of action.…”

Section: Contextual Multi-armed Banditsmentioning

confidence: 99%

Putting bandits into context: How function learning supports decision making

Schulz

¹

,

Konstantinidis

²

,

Speekenbrink

³

2016

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce the contextual multi-armed bandit task as a framework to investigate learning and decision making in uncertain environments. In this novel paradigm, participants repeatedly choose between multiple options in order to maximise their rewards. The options are described by a number of contextual features which are predictive of the rewards through initially unknown functions. From their experience with choosing options and observing the consequences of their decisions, participants can learn about the functional relation between contexts and rewards and improve their decision strategy over time. In three experiments, we explore participants' behaviour in such learning environments. We predict participants' behaviour by context-blind (mean-tracking, Kalman filter) and contextual (Gaussian process and linear regression) learning approaches combined with different choice strategies. Participants are mostly able to learn about the context-reward functions and their behaviour is best described by a Gaussian process learning strategy which generalizes previous experience to similar instances. In a relatively simple task with binary features, they seem to combine this learning with a "probability of improvement" decision strategy which focuses on alternatives that are expected to lead to an improvement upon a current favourite option. In a task with continuous features that are linearly related to the rewards, participants seem to more explicitly balance exploration and exploitation. Finally, in a difficult learning environment where the relation between features and rewards is non-linear, some participants are again well-described by a Gaussian process learning strategy, whereas others revert to context-blind strategies.

show abstract

“…Participants rarely converge to selecting the same option exclusively. Specifically, while participants in DF tasks often learn to select options of higher expected value (EV) more frequently with experience (Ashby & Rakow, 2015;Jessup, Bishara, & Busemeyer, 2008), some variability in choice remains even after a great deal of experience (Konstantinidis, Ashby, & Gonzalez, 2015), and even occurs when options are fully described (Barron & Lieder, 2010; Lejaraga & Gonzallez, 2011; Weiss-Cohen, Konstantinidis, Speekenbrink, & Harvey, 2016;Yechiam, Barron, & Erev, 2005). As noted above, this pattern was observed in probability learning tasks (Bereby-Meyer & Erev, 1998;Erev, Bereby-Meyer, & Roth, 1999) and in a DF task where one option deterministically dominated another option: Variance in choice remained after 200 trials of choosing between one option providing 11 points with certainty and another providing 10 points with certainty (Haruvy & Erev, 2002).…”

Section: Takedownmentioning

confidence: 99%

Choice in experiential learning: True preferences or experimental artifacts?

Ashby

¹

,

Konstantinidis

²

,

Yechiam

³

2017

Self Cite

View full text Add to dashboard Cite

The rate of selecting different options in the decisions-from-feedback paradigm is commonly used to measure preferences resulting from experiential learning. While convergence to a single option increases with experience, some variance in choice remains even when options are static and offer fixed rewards. Employing a decisions-from-feedback paradigm followed by a policy-setting task, we examined whether the observed variance in choice is driven by factors related to the paradigm itself: Continued exploration (e.g., believing options are non-stationary) or exploitation of perceived outcome patterns (i.e., a belief that sequential choices are not independent). Across two studies, participants showed variance in their choices, which was related (i.e., proportional) to the policies they set. In addition, in Study 2, participants' reported under-confidence was associated with the amount of choice variance in later choices and policies. These results suggest that variance in choice is better explained by participants lacking confidence in knowing which option is better, rather than methodological artifacts (i.e., exploration or failures to recognize outcome independence). As such, the current studies provide evidence for the decisions-from-feedback paradigm's validity as a behavioral research method for assessing learned preferences.

show abstract

Incorporating conflicting descriptions into decisions from experience

Cited by 43 publications

References 43 publications

Putting bandits into context: How function learning supports decision making.

Putting bandits into context: How function learning supports decision making.

Putting bandits into context: How function learning supports decision making

Choice in experiential learning: True preferences or experimental artifacts?

Contact Info

Product

Resources

About