Self-modification and mortality in artificial agents

  • Authors:
  • Laurent Orseau;Mark Ring

  • Affiliations:
  • UMR AgroParisTech, INRA, Paris, France;IDSIA, University of Lugano, SUPSI, Manno-Lugano, Switzerland

  • Venue:
  • AGI'11 Proceedings of the 4th international conference on Artificial general intelligence
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper considers the consequences of endowing an intelligent agent with the ability to modify its own code. The intelligent agent is patterned closely after AIXI [1], but the environment has read-only access to the agent's description. On the basis of some simple modifications to the utility and horizon functions, we are able to discuss and compare some very different kinds of agents, specifically: reinforcement-learning, goal-seeking, predictive, and knowledge-seeking agents. In particular, we introduce what we call the "Simpleton Gambit" which allows us to discuss whether these agents would choose to modify themselves toward their own detriment.