Abstract-In this paper we introduce a novel method for discovery of value functions for Markov Decision Processes (MDPs). This method, which we call Value Function Discovery (VFD), is based on ideas from the Evolutionary Algorithm field. VFD's key feature is that it discovers descriptions of value functions that are algebraic in nature. This feature is unique, because the descriptions include the model parameters of the MDP. The algebraic expression of the value function discovered by VFD can be used in several scenarios, e.g., conversion to a policy (with one-step policy improvement) or control of systems with time-varying parameters. The work in this paper is a first step towards exploring potential usage scenarios of discovered value functions. We give a detailed description of VFD and illustrate its application on an example MDP. For this MDP we let VFD discover an algebraic description of a value function that closely resembles the optimal value function. The discovered value function is then used to obtain a policy, which we compare numerically to the optimal policy of the MDP. The resulting policy shows near-optimal performance on a wide range of model parameters. Finally, we identify and discuss future application scenarios of discovered value functions.

Sensors are increasingly part of our daily lives: motion detection, lighting control, and energy consumption all rely on sensors. Combining this information into, for instance, simple and comprehensive graphs can be quite challenging. Dimensionality reduction is often used to address this problem, by decreasing the number of variables in the data and looking for shorter representations. However, dimensionality reduction is often aimed at normal daily data, and applying it to events deviating from this daily data (so-called outliers) can affect such events negatively. In particular, outliers might go unnoticed.In this paper we show that dimensionality reduction can indeed have a large impact on outliers. To that end we apply three dimensionality reduction techniques to three real-world data sets, and inspect how well they preserve outliers. We use several performance measures to show how well these techniques are capable of preserving outliers, and we discuss the results.1

a b s t r a c tIncreasing computing capabilities of modern sensors have enabled current wireless sensor networks to process queries within the network. This complements the traditional features of the sensor networks such as sensing the environment and communicating the data. Query processing, however, poses Quality of Service challenges such as query waiting time and validity (age) of the data. We focus on the processing cost of queries as a trade-off between the time queries wait to be processed and the age of the data provided to the queries. To model this trade-off, we propose a Continuous Time Markov Decision Process which assigns queries either to the sensor network, where queries wait to be processed, or to a central database, which provides stored and possibly outdated data. To compute an optimal query assignment policy, a Discrete Time Markov Decision Process, shown to be stochastically equivalent to the initial continuous time process, is formulated. A comparative numerical analysis of the performance of the optimal assignment policy and of several heuristics, derived from practice, is performed. This provides a theoretical support for the design and implementation of WSN applications, while ensuring a close-tooptimum performance of the system.

We investigate control of a queueing system in which a component of the state space is subject to aging. The controller can choose to forward incoming queries to the system (where it needs time for processing), or respond with a previously generated response (incurring a penalty for not providing a fresh value). Hence, the controller faces a trade-off between data freshness and response times. We model the system as a complex Markov Decision Process, simplify it, and construct a control policy. This policy shows near-optimal performance and achieves lower costs than both a myopic policy and a threshold policy.