Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances

Imitation Learning - Learning paradigm in which an agent seeks to acquire a policy by observing and imitating the behavior of $1\dots N$ proficient agents, referred to as the expert

In Imitation Learning the expert provides Dataset D of expert-related behavioral data

$$\begin{array}{r}M\in {\mathbb{N}}^{+}\\ D=\{(s_i,a_i){\}}_{i=1}^M\\ D=\{(s_i,a_i,s_i^{\prime }{\}}_{i=1}^M\\ D=\{(s_i,s_i^{\prime }{\}}_{i=1}^M\end{array}$$

In english: The format of the input the agent can learn from

• M is a positive number
• In the state $s_i$ the expert took action $a_i$
• In the state $s_i$ the expert took action $a_i$ and the resulting state was $s_i^{\prime }$
• pre-action state $s_i$ and post-action state $s_i^{\prime }$

$${\pi }_e:S\to △(A)$$

In english: For a specific state the probability distribution of all the possible actions

• S = the state space (set of all possible states)
• A = the action space (set of all possible actions)
• $△(A)$ = the set of probability distributions over actions in A

Explicit Imitation - The Dataset provided only contains state-action pairs $\{(s_i,a_i{\}}_{i=1}^M$

Implicit Imitation - The Dataset ONLY contains state transitions $\{(s_i){\}}_{i=1}^M$