Motivated by (approximate) dynamic programming and model predictive control problems, we analyse the stability of deterministic nonlinear discrete-time systems whose inputs minimize a discounted finite-horizon cost. We assume that the system satisfies stabilizability and detectability properties with respect to the stage cost. Then, a Lyapunov function for the closedloop system is constructed and a uniform semiglobal stability property is ensured, where the adjustable parameters are both the discount factor and the horizon length, which corresponds to the number of iterations for dynamic programming algorithms like value iteration. Stronger stability properties such as global exponential stability are also provided by strengthening the initial assumptions. We give bounds on the discount factor and the horizon length under which stability holds and we provide conditions under which these are less conservative than the bounds of the literature for discounted infinite-horizon cost and undiscounted finite-horizon costs, respectively. In addition, we provide new relationships between the optimal value functions of the discounted, undiscounted, infinite-horizon and finite-horizon costs respectively, which are very different from those available in the approximate dynamic programming literature. These relationships rely on assumptions that are more likely to be satisfied in a control context. Finally, we investigate stability when only a near-optimal sequence of inputs for the discounted finitehorizon cost is available, covering approximate value iteration as a particular case.

Discounted costs are considered in many fields, like reinforcement learning, for which various algorithms can be used to obtain optimal inputs for finite horizons. The related literature mostly concentrates on optimality and largely ignores stability. In this context, we study stability of general nonlinear discrete-time systems controlled by an optimal sequence of inputs that minimizes a finite-horizon discounted cost computed in a receding horizon fashion. Assumptions are made related to the stabilizability of the system and its detectability with respect to the stage cost. Then, a Lyapunov function for the closedloop system with the receding horizon controller is constructed and a uniform semiglobal stability property is ensured, where the adjustable parameters are both the discount factor and the horizon length. Uniform global exponential stability is guaranteed by strengthening the initial assumptions, in which case explicit bounds on the discount factor and the horizon length are provided. We compare the obtained bounds in the particular cases where there is no discount or the horizon is infinite, respectively, with related results in the literature and we show our bounds improve existing ones on the examples considered.

Originating in the artificial intelligence literature, optimistic planning (OP) is an algorithm that generates nearoptimal control inputs for generic nonlinear discrete-time systems whose input set is finite. This technique is therefore relevant for the near-optimal control of nonlinear switched systems, for which the switching signal is the control. However, OP exhibits several limitations, which prevent its application in a standard control context. First, it requires the stage cost to take values in [0,1], an unnatural prerequisite as it excludes, for instance, quadratic stage costs. Second, it requires the cost function to be discounted. Third, it applies for reward maximization, and not cost minimization. In this paper, we modify OP to overcome these limitations, and we call the new algorithm OPmin. We then make stabilizability and detectability assumptions, under which we derive nearoptimality guarantees for OPmin and we show that the obtained bound has major advantages compared to the bound originally given by OP. In addition, we prove that a system whose inputs are generated by OPmin in a receding-horizon fashion exhibits stability properties. As a result, OPmin provides a new tool for the near-optimal, stable control of nonlinear switched discretetime systems for generic cost functions.

No abstract

Value iteration is a method to generate optimal control inputs for generic nonlinear systems and cost functions. Its implementation typically leads to approximation errors, which may have a major impact on the closed-loop system performance. We talk in this case of approximate value iteration (AVI). In this paper, we investigate the stability of systems for which the inputs are obtained by AVI. We consider deterministic discrete-time nonlinear plants and a class of general, possibly discounted, costs. We model the closed-loop system as a family of systems parameterized by tunable parameters, which are used for the approximation of the value function at different iterations, the discount factor and the iteration step at which we stop running the algorithm. It is shown, under natural stabilizability and detectability properties as well as mild conditions on the approximation errors, that the family of closed-loop systems exhibit local practical stability properties. The analysis is based on the construction of a Lyapunov function given by the sum of the approximate value function and the Lyapunov-like function that characterizes the detectability of the system. By strengthening our conditions, asymptotic and exponential stability properties are guaranteed.