Improving Bayesian Reinforcement Learning Using Transition Abstraction

• Authors: Daniel Acuña, Paul Schrater
• Year: 2009
• Abstract: Bayesian Reinforcement Learning (BRL) provides an optimal solution to on-line learning while acting, but it is computationally intractable for all but the simplest problems: at each decision time, an agent should weigh ll possible courses of actions by beliefs about future outcomes constructed over long time horizons. To improve tractability, previous research has focused on sparsely sampling possible courses of action that are most relevant to computing value; however, sampling alone does not scale well to learnger environments. In this paper, we investigate whether an abstraction called projects—parts of the transition dynamics that bias the look ahead to areas of the environment that are promising—can scale up BRL. We modify a sparse sampler to incorporate projects. We test our algorithm on standard problems that require effective exploration–exploitation balance and show that learning can be significantly sped up compared to a simpler BRL and classic Q-learning.
• Conference proceedings: In Proceddings of the ICML/UIA/COLT Workshop on Abstraction in Reinforcement Learning. Montreal, Canada, 2009.
• Files: Paper [PDF]

The views and opinions expressed in this page are strictly those of the page author.
The contents of this page have not been reviewed or approved by the University of Minnesota.