Proximal Policy Optimization

Proximal Policy Optimization is that we want improve training stability of the policy by limiting the change you make to the policy at each training epoch

2 reasons behind this:

• Smaller policy changes = more likely to converge to optimal solution
• Too big step in a policy update can result in falling “off the cliff” (getting a bad policy) and having a long time and even no possibility to recover

PPO = update policy conservatively, this is done by measuring how much policy changed to the former using a ratio calculation between current and former policy, clip this ratio in a range $[1-ϵ,1+ϵ]$

Function made to avoid destructive large weight updates:

$$\begin{array}{c}{L}^{CLIP}(\theta )={\hat{\mathbb{E}}}_t\left[\min (r_t(\theta ){\hat{A}}_t,\text{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t)\right]\end{array}$$

Ratio function: $r_t(\theta )=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\text{old}}(a_t\mid s_t)}}$

• if $r_t(\theta )>1$ the action $a_t$ at the state $s_t$ is in current policy than the old one
• if $r_t(\theta )$ is between 0 and 1, the action is less likely for current policy than for old one