This paper presents a dynamic fuzzy Q-learning (DFQL) method that is capable of tuning fuzzy inference systems (FIS) online. A novel online self-organizing learning algorithm is developed so that structure and parameters identification are accomplished automatically and simultaneously based only on Q-learning. Self-organizing fuzzy inference is introduced to calculate actions and Q-functions so as to enable us to deal with continuous-valued states and actions. Fuzzy rules provide a natural mean of incorporating the bias components for rapid reinforcement learning. Experimental results and comparative studies with the fuzzy Q-learning (FQL) and continuous-action Q-learning in the wall-following task of mobile robots demonstrate that the proposed DFQL method is superior.

No abstract

Oxygen distribution and uptake in the hyporheic zone regulate various redox-sensitive reactions and influence habitat conditions. Despite the fact that fine-grain sediments in streams and rivers are commonly in motion, most studies on biogeochemistry have focused on stagnant sediments. In order to evaluate the effect of bed form celerity on oxygen dynamics and uptake in sandy beds, we conducted experiments in a recirculating indoor flume. Oxygen distribution in the bed was measured under various celerities using 2D planar optodes. Bed morphodynamics were measured by a surface elevation sensor and time-lapse photography. Oxygenated zones in stationary beds had a conchoidal shape due to influx through the stoss side of the bed form, and upwelling anoxic water at the lee side. Increasing bed celerity resulted in the gradual disappearance of the upwelling anoxic zone and flattening of the interface between the oxic (moving fraction of the bed) and the anoxic zone (stationary fraction of the bed), as well as in a reduction of the volumetric oxygen uptake rates due shortened residence times in the hyporheic zone. These results suggest that including processes related to bed form migration are important for understanding the biogeochemistry of hyporheic zones.

Abstract. Hydrological model parameters play an important role in the ability of model prediction. In a stationary context, parameters of hydrological models are treated as constants; however, model parameters may vary with time under climate change and anthropogenic activities. The technique of ensemble Kalman filter (EnKF) is proposed to identify the temporal variation of parameters for a two-parameter monthly water balance model (TWBM) by assimilating the runoff observations. Through a synthetic experiment, the proposed method is evaluated with time-invariant (i.e., constant) parameters and different types of parameter variations, including trend, abrupt change and periodicity. Various levels of observation uncertainty are designed to examine the performance of the EnKF. The results show that the EnKF can successfully capture the temporal variations of the model parameters. The application to the Wudinghe basin shows that the water storage capacity (SC) of the TWBM model has an apparent increasing trend during the period from 1958 to 2000. The identified temporal variation of SC is explained by land use and land cover changes due to soil and water conservation measures. In contrast, the application to the Tongtianhe basin shows that the estimated SC has no significant variation during the simulation period of 1982-2013, corresponding to the relatively stationary catchment properties. The evapotranspiration parameter (C) has temporal variations while no obvious change patterns exist. The proposed method provides an effective tool for quantifying the temporal variations of the model parameters, thereby improving the accuracy and reliability of model simulations and forecasts.