Uncertainty quantification and exploration–exploitation trade-off in humans

Abstract

The main objective of this paper is to outline a theoretical framework to analyse how humans' decision-making strategies under uncertainty manage the trade-off between information gathering (exploration) and reward seeking (exploitation). A key observation, motivating this line of research, is the awareness that human learners are amazingly fast and effective at adapting to unfamiliar environments and incorporating upcoming knowledge: this is an intriguing behaviour for cognitive sciences as well as an important challenge for Machine Learning. The target problem considered is active learning in a black-box optimization task and more specifically how the exploration/exploitation dilemma can be modelled within Gaussian Process based Bayesian Optimization framework, which is in turn based on uncertainty quantification. The main contribution is to analyse humans' decisions with respect to Pareto rationality where the two objectives are improvement expected and uncertainty quantification. According to this Pareto rationality model, if a decision set contains a Pareto efficient (dominant) strategy, a rational decision maker should always select the dominant strategy over its dominated alternatives. The distance from the Pareto frontier determines whether a choice is (Pareto) rational (i.e., lays on the frontier) or is associated to "exasperate" exploration. However, since the uncertainty is one of the two objectives defining the Pareto frontier, we have investigated three different uncertainty quantification measures and selected the one resulting more compliant with the Pareto rationality model proposed. The key result is an analytical framework to characterize how deviations from "rationality" depend on uncertainty quantifications and the evolution of the reward seeking process.

On the use of Wasserstein distance in the distributional analysis of human decision making under uncertainty

“…The only way to analyse how different uncertainty quantification measures can lead to completely different decisions -even if anyway Pareto rational -is to localize, within the search space Ω ⊂ ℜ d , the locations whose associated objectives lay on the Pareto frontier (namely, the Pareto set). According to results reported in [41], the Pareto-rational decisions (i.e., Pareto set) do not significantly depend on kernel. Instead, an evident difference arises with respect to the uncertainty quantification measure: one of the three considered in the study allows for accounting, as Pareto rational, choices which are instead explorative for the other two measures.…”

“…The most relevant result [41] is that, in some cases, it is possible to observe a shift from Pareto-rationality to not-Pareto-rationality, whichever is the uncertainty quantification measure adopted, including that maximizing the number of Pareto-rational decisions. This means that, in the case where there is not evident chance to exploit, and there is not any explorationexploitation trade-off compliant to the Pareto-rational model, humans move towards "exasperated" exploration, where with the term "exasperated" we want to remark the fact that the decision is even more explorative than the pure exploration offered by the Pareto-rational model.…”

Wasserstein enabled Bayesian optimization of composite functions

“…The acquisition function manages the balance between exploration and exploitation, it is the key driver of the sample efficiency of BO and is an important concept also outside machine learning (Candelieri et al 2021). It drives the search of the new evaluation points towards regions of the search space with potential better values of the objective function either because value of ( ) is better or the uncertainty represented by 2 ( ) is high (or both).…”