Elementary results on solutions to the bellman equation of dynamic programming: existence, uniqueness, and convergence

Abstract

We establish some elementary results on solutions to the Bellman equation without introducing any topological assumption. Under a small number of conditions, we show that the Bellman equation has a unique solution in a certain set, that this solution is the value function, and that the value function can be computed by value iteration with an appropriate initial condition. In addition, we show that the value function can be computed by the same procedure under alternative conditions. We apply our results to two optimal growth models: one with a discontinuous production function and the other with "roughly increasing" returns.

An order-theoretic approach to dynamic programming: an exposition

“…It can be shown that given any fixed point v of B in [v, v], if v satisfies (6.7), then v ≥ v * , and if v satisfies (6.8), then v ≤ v * ; see Kamihigashi (2013) and Stokey and Lucas (1989, Theorem 4.3). Adding these requirements to Proposition 5.2 and recalling footnote 5, we obtain parts (a) and (b) of the following result.…”

“…Recently, Kamihigashi (2013) obtained a simple yet useful result on the existence, uniqueness, and stability of a solution to the Bellman equation, i.e., a fixed point of the Bellman operator. In this note, we present the logic behind this result as well as explain why the order-theoretic approach used in Kamihigashi (2013) is useful but not sufficient to characterize the value function. To present our arguments in precise terms, we start by introducing some definitions and notations.…”

“…In particular, we explain how order-theoretic fixed point theorems can be used to establish the existence of a fixed point of the Bellman operator, as well as why they are not sufficient to characterize the value function. By doing this, we present the logic behind the simple yet useful result recently obtained by Kamihigashi (2013) based on this order-theoretic approach. …”

Infinite-horizon deterministic dynamic programming in discrete time: a monotone convergence principle and a penalty method

“…[10][11][12] This approach can be viewed as an extension of the earlier order-theoretic approach of [13,Chapter 5]. One of the results based on the new approach is the following [10, Theorem 2.2]: value iteration converges increasingly to the value function if the initial function is dominated by the value function, is mapped upward by the Bellman operator and satisfies a transversality-like condition.…”

“…We show that the value function is a fixed point of this modified Bellman operator. The monotone convergence principle is that value iteration converges increasingly to the value function if the initial function is dominated by the value function, is mapped upward by the modified Bellman operator and satisfies the same transversality-like condition as in the result of [10,Theorem 2.2].…”