Recursive Adaptation of Stepsize Parameter for Non-stationary Environments

Authors:
Itsuki Noda
Affiliations:
Information Technology Research Institute, National Institute of Advanced Industrial Science and Technology, Tsukuba, Japan 305-8568 and School of Information Science, Japan Advanced Institute of ...
Venue:
PRIMA '09 Proceedings of the 12th International Conference on Principles of Practice in Multi-Agent Systems
Year:
2009

Citing 5
Cited 2

Introduction to Reinforcement Learning

Introduction to Reinforcement Learning
Speeding-up Reinforcement Learning with Multi-step Actions

ICANN '02 Proceedings of the International Conference on Artificial Neural Networks
Learning Rates for Q-learning

The Journal of Machine Learning Research
Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming

Machine Learning
IFSA: incremental feature-set augmentation for reinforcement learning tasks

Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems

Fixed point method for autonomous on-line neural network training

Neurocomputing
2013 Special Issue: Autonomous reinforcement learning with experience replay

Neural Networks

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this article, we propose a method to adapt stepsize parameters used in reinforcement learning for non-stationary environments. When the environment is non-stationary, the learning agent must adapt learning parameters like stepsize to the changes of environment through continuous learning. We show several theorems on higher-order derivatives of exponential moving average, which is a base schema of major reinforcement learning methods, using stepsize parameters. We also derive a systematic mechanism to calculate these derivatives in a recursive manner. Based on it, we construct a precise and flexible adaptation method for the stepsize parameter in order to maximize a certain criterion. The proposed method is also validated by several experimental results.