Value-difference based exploration: adaptive control between epsilon-greedy and softmax

Authors:
Michel Tokic;Günther Palm
Affiliations:
Institute of Neural Information Processing, University of Ulm, Ulm and Institute of Applied Research, University of Applied Sciences, Weingarten, Germany;Institute of Neural Information Processing, University of Ulm, Ulm, Germany
Venue:
KI'11 Proceedings of the 34th Annual German conference on Advances in artificial intelligence
Year:
2011

Citing 8
Cited 4

Technical Note: \cal Q-Learning

Machine Learning
Learning in embedded systems

Learning in embedded systems
Introduction to Reinforcement Learning

Introduction to Reinforcement Learning
Efficient Exploration In Reinforcement Learning

Efficient Exploration In Reinforcement Learning
Using confidence bounds for exploitation-exploration trade-offs

The Journal of Machine Learning Research
Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming

Machine Learning
Adaptive ε-greedy exploration in reinforcement learning based on value differences

KI'10 Proceedings of the 33rd annual German conference on Advances in artificial intelligence
Multi-armed bandit algorithms and empirical evaluation

ECML'05 Proceedings of the 16th European conference on Machine Learning

Adaptive exploration using stochastic neurons

ICANN'12 Proceedings of the 22nd international conference on Artificial Neural Networks and Machine Learning - Volume Part II
Gradient algorithms for exploration/exploitation trade-offs: global and local variants

ANNPR'12 Proceedings of the 5th INNS IAPR TC 3 GIRPR conference on Artificial Neural Networks in Pattern Recognition
A reinforcement learning based solution for cognitive network cooperation between co-located, heterogeneous wireless sensor networks

Ad Hoc Networks
GRASS: trimming stragglers in approximation analytics

NSDI'14 Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper proposes "Value-Difference Based Exploration combined with Softmax action selection" (VDBE-Softmax) as an adaptive exploration/exploitation policy for temporal-difference learning. The advantage of the proposed approach is that exploration actions are only selected in situations when the knowledge about the environment is uncertain, which is indicated by fluctuating values during learning. The method is evaluated in experiments having deterministic rewards and a mixture of both deterministic and stochastic rewards. The results show that a VDBE-Softmax policy can outperform ε-greedy, Softmax and VDBE policies in combination with on- and off-policy learning algorithms such as Q-learning and Sarsa. Furthermore, it is also shown that VDBE-Softmax is more reliable in case of value-function oscillations.