Self-organizing state aggregation for architecture design of Q-learning

  • Authors:

  • Kao-Shing Hwang;Hsin-Yi Lin;Yuan-Pao Hsu;Hung-Hsiu Yu

  • Affiliations:

  • Department of Electrical Engineering, National Chung-Cheng University, Chiayi 711, Taiwan and Department of Electrical Engineering, National Sun Yat-sen University, Kaohisung 800, Taiwan;Robotics Control Department, Intelligent Robotics Technology Division, Mechanical and Systems Research Laboratories (MSL), Industrial Technology Research Institute (ITRI), Hsinchu 310, Taiwan;Department of Computer Science and Information Engineering, National Formosa University, Yunlin 632, Taiwan;Robotics Control Department, Intelligent Robotics Technology Division, Mechanical and Systems Research Laboratories (MSL), Industrial Technology Research Institute (ITRI), Hsinchu 310, Taiwan

  • Venue:
  • Information Sciences: an International Journal
  • Year:
  • 2011

Quantified Score

Hi-index 0.07

Visualization

Abstract

This work describes a novel algorithm that integrates an adaptive resonance method (ARM), i.e. an ART-based algorithm with a self-organized design, and a Q-learning algorithm. By dynamically adjusting the size of sensitivity regions of each neuron and adaptively eliminating one of the redundant neurons, ARM can preserve resources, i.e. available neurons, to accommodate additional categories. As a dynamic programming-based reinforcement learning method, Q-learning involves use of the learned action-value function, Q, which directly approximates Q^*, i.e. the optimal action-value function, which is independent of the policy followed. In the proposed algorithm, ARM functions as a cluster to classify input vectors from the outside world. Clustered results are then sent to the Q-learning design in order to learn how to implement the optimum actions to the outside world. Simulation results of the well-known control algorithm of balancing an inverted pendulum on a cart demonstrates the effectiveness of the proposed algorithm.