Header logo is ei

Apprenticeship Learning with Few Examples

2013

Article

ei


We consider the problem of imitation learning when the examples, provided by an expert human, are scarce. Apprenticeship learning via inverse reinforcement learning provides an efficient tool for generalizing the examples, based on the assumption that the expert's policy maximizes a value function, which is a linear combination of state and action features. Most apprenticeship learning algorithms use only simple empirical averages of the features in the demonstrations as a statistics of the expert's policy. However, this method is efficient only when the number of examples is sufficiently large to cover most of the states, or the dynamics of the system is nearly deterministic. In this paper, we show that the quality of the learned policies is sensitive to the error in estimating the averages of the features when the dynamics of the system is stochastic. To reduce this error, we introduce two new approaches for bootstrapping the demonstrations by assuming that the expert is near-optimal and the dynamics of the system is known. In the first approach, the expert's examples are used to learn a reward function and to generate furthermore examples from the corresponding optimal policy. The second approach uses a transfer technique, known as graph homomorphism, in order to generalize the expert's actions to unvisited regions of the state space. Empirical results on simulated robot navigation problems show that our approach is able to learn sufficiently good policies from a significantly small number of examples.

Author(s): Boularias, A. and Chaib-draa, B.
Journal: Neurocomputing
Volume: 104
Pages: 83-96
Year: 2013
Month: March
Day: 0

Department(s): Empirical Inference
Bibtex Type: Article (article)

Digital: 0
DOI: 10.1016/j.neucom.2012.11.002

Links: Web

BibTex

@article{BoulariasC2012,
  title = {Apprenticeship Learning with Few Examples},
  author = {Boularias, A. and Chaib-draa, B.},
  journal = {Neurocomputing},
  volume = {104},
  pages = {83-96},
  month = mar,
  year = {2013},
  month_numeric = {3}
}