Reinforcement Learning: Proximal Policy Optimization (PPO)

After exploring trust region methods, we arrived at an important realization.

Policy gradients are powerful, but they can also be dangerously unstable.

A single gradient step might change the policy so drastically that the agent suddenly behaves like a completely different system. Trust Region Policy Optimization (TRPO) solved this problem by introducing a constraint: every update must keep the new policy close to the old one, measured through KL divergence.

In theory, that idea is beautiful.

But in practice, TRPO is quite heavy. The algorithm requires approximating second-order derivatives, computing Fisher information matrices, and solving constrained optimization problems with specialized numerical methods.

Do we really need such complicated machinery to keep policy updates under control?

Or is there a simpler way to prevent policies from changing too much?

To see how that question led to PPO, let’s revisit something that appeared earlier in policy gradients but didn’t seem very important at the time.

When we derived the policy gradient, we wrote the update like this:

The update increases the probability of actions with positive advantage and decreases the probability of actions with negative advantage.

But if we look at it slightly differently, something interesting appears.

The expectation is taken over trajectories generated by the policy $\pi_\theta$.

When we derived the gradient, we eventually reached

This expression is subtle but extremely important.

The expectation is still under the distribution of trajectories produced by the current policy $\pi_\theta$. That means when we approximate this expectation with samples, those samples must come from that same policy. This is why the original REINFORCE algorithm behaves the way it does.

Run the environment using policy $\pi_\theta$

Collect trajectories

Compute returns or advantages

Perform a gradient update

Discard the data

Run the environment again with the updated policy

Every update requires fresh trajectories, because after the parameters change, the policy changes, and the old trajectories are no longer drawn from the correct distribution.

This is what makes vanilla policy gradient methods strictly on-policy.

But if we pause for a moment, something feels wasteful.

Imagine the agent just collected $10{,}000$ steps of experience. We compute a gradient update, slightly adjust the neural network parameters, and then immediately throw away all that data.

Even though the policy changed only a little.

This feels like reading an entire book, making a tiny correction to your understanding, and then deciding you must reread the whole book again before learning anything more.

So a natural question appears:

Can we reuse the same trajectories for multiple optimization steps?

At first this sounds harmless. If the policy only changes slightly, those trajectories should still contain useful information.

But mathematically there is a problem.

Suppose the trajectories were collected using the old policy

But during optimization we evaluate the gradient using a new policy

If we directly plug those trajectories into the estimator

we are actually estimating

instead of

The distribution is wrong.

This situation should feel familiar.

In the off-policy Monte Carlo setting, we faced the same mismatch. There we had data generated by a behavior policy $b$, but we wanted to evaluate a target policy $\pi$.

The fix was importance sampling.

We multiplied each sample by the ratio

which corrected the distribution difference.

The same trick works here.

Our behavior policy is the one that generated the trajectories:

And our target policy is the one we are optimizing:

So the correction factor becomes

This ratio answers a simple question:

How much more (or less) likely would the new policy choose this action compared to the old policy?

If the ratio equals $1$, both policies behave the same.

If the ratio is greater than $1$, the new policy prefers that action more strongly.

If the ratio is less than $1$, the new policy prefers it less.

Using importance ratios, we can form a surrogate objective on data collected by the old policy:

Now something important becomes possible.

We can collect trajectories using the old policy $\pi_{\theta_{\text{old}}}$ and then perform multiple gradient updates using those same trajectories, because the ratio corrects the distribution mismatch.

In other words, instead of this inefficient loop

collect data → update once → discard data

we can do something better:

collect data → update several times → then collect new data.

This dramatically improves sample efficiency, especially in environments where generating experience is expensive.

When Importance Ratios Become Dangerous

But now another problem appears.

If the new policy becomes very different from the old policy, the ratio

can grow extremely large.

This should again feel familiar from the off-policy Monte Carlo setting. There we saw that importance sampling ratios could explode when the target policy behaved very differently from the behavior policy.

Exactly the same instability appears here.

One gradient update might push the probability of an action dramatically higher, making the ratio huge. The next update then becomes wildly unstable.

This is precisely the problem that Trust Region methods tried to control using KL divergence constraints.

Proximal Policy Optimization (PPO) takes a simpler approach.

Instead of solving a complicated constrained optimization problem, PPO simply limits how large the probability ratio is allowed to become.

If the optimizer tries to increase the probability of an action too aggressively, we clip the ratio so that the objective stops improving beyond a certain point.

This produces the famous PPO objective:

The clipping operation keeps the ratio within a small window around $1$.

If

the ratio is clipped to $1+\epsilon$.

If

it is clipped to $1-\epsilon$.

This simple trick creates a soft trust region. The policy can still improve, but once the change becomes too large, the objective stops encouraging further movement.

In effect, PPO captures the spirit of trust region optimization without the heavy machinery of TRPO.

And this is the core idea behind PPO.

We reuse trajectories to improve sample efficiency, correct the distribution mismatch using importance sampling ratios, and prevent unstable updates by clipping those ratios.

With just these ingredients advantages, probability ratios, and clipping, PPO manages to achieve the stability of trust region methods while remaining remarkably simple to implement.

Single Update

To really understand what PPO is doing, it helps to slow down and watch how a single update actually happens.

Imagine a simple situation. The agent visits some state $s$ and takes action $a$. Under the old policy that generated the trajectory, the probability of that action was

Later, during optimization, the neural network parameters have shifted slightly. Under the new parameters $\theta$, the same action now has probability

So the probability ratio becomes

Now suppose the advantage for that state–action pair is positive:

This means the action performed better than expected. The policy should increase its probability.

If we plug this into the PPO objective, the first term becomes

But PPO also computes the clipped version. Suppose the clipping threshold is

Then the allowed range for the ratio is

Our ratio $1.30$ lies outside that range, so it gets clipped:

So the clipped objective becomes

And PPO takes the minimum of the two:

Notice what just happened.

Even though the optimizer tried to push the probability much higher, the objective refuses to reward that extra increase. Once the ratio exceeds the allowed window, the improvement simply stops growing.

How PPO Translates Into a Policy Update

But the PPO objective ultimately serves a familiar purpose: producing a gradient that updates the policy parameters.

In earlier policy gradient methods, the actor update looked like this:

The gradient term

tells us how to adjust the neural network so that the probability of the chosen action changes. The advantage $A_t$ tells us whether that probability should go up or down. When we introduced Generalized Advantage Estimation (GAE) earlier, we replaced the noisy advantage with a smoother estimate

so the update became

Now PPO introduces the probability ratio

which measures how much the new policy deviates from the old one. Instead of directly scaling the gradient by $A_t^{\text{GAE}}$, PPO uses the clipped objective

So the actor update becomes

and this is the first place where PPO looks a little different from the policy-gradient update we were using before. Earlier, the update was written in the familiar form

This form is very intuitive. The term

tells us how to move the policy, and the advantage tells us whether we should move in that direction strongly, weakly, positively, or negatively.

Now PPO does not begin by writing the update directly in that form. Instead, it begins with a new objective built for reused data:

where

At first glance, it may feel like

has disappeared. But it has not really disappeared. It is simply hiding inside the gradient of the ratio. To see this, ignore clipping for a moment and look only at the simpler surrogate term

If we differentiate it, we obtain

Now remember that during one PPO update epoch, the old policy is frozen. So

is just a constant with respect to $\theta$. That means

Now use the identity

and substitute it in:

So the gradient becomes

Now the connection becomes clear.

In vanilla policy gradient, the update contribution of a sample was

In PPO before clipping, that same contribution becomes

So we do not remove the log term. We keep it. PPO simply inserts an extra multiplier, the probability ratio because we are optimizing on data collected by an older policy. Now clipping adds the safety mechanism. Instead of always trusting

we compare it against the clipped version and take the smaller one:

So the full PPO actor update is written as

But when that gradient is expanded, it still contains

The only difference is that now the sample is weighted by the ratio, and that weight is prevented from becoming too extreme by clipping.

So if we want to write PPO in a form that still resembles the earlier actor update, the unclipped intuition is

Then PPO adds the rule: this update is allowed only as long as the ratio stays in a safe range. If the optimizer tries to push the action probability too far away from the old policy, the clipped objective stops rewarding that movement.

That is the real transition from vanilla actor–critic to PPO. The log-gradient remains the engine that changes the policy. The ratio tells us how far we have drifted from the behavior policy. And clipping acts like a guardrail that says: learn from this sample, but do not overreact to it.

This is why PPO is called “proximal” policy optimization. The word proximal simply means nearby. The algorithm encourages the new policy to remain close to the old one while still improving performance.

But now we should make one subtle correction, because this is exactly the place where PPO often becomes confusing.

When we write

we are speaking in the language of gradient ascent. We are saying: “here is an objective we want to maximize.” Mathematically, That is fine.

But in actual deep-learning code, optimizers like Adam are almost always written to minimize a loss. So PPO is usually implemented not by maximizing $L_{\text{clip}}$, but by minimizing the negative of it. That gives the clipped actor loss:

Here I am writing the advantage as $\hat A_t$, because in practice this is usually the estimated advantage, often GAE.

This minus sign matters. Without it, it is easy to mix up whether the optimizer is being asked to increase or decrease the objective. So the most precise statement is this:

PPO maximizes the clipped surrogate objective,

or equivalently,

PPO minimizes the negative clipped actor loss.

That is the corrected bridge between the elegant math and the code people actually run.

Now Someone has to estimate those advantages. Someone has to say whether the outcome of an action was better or worse than expected. That is the critic’s job.

If the actor is the part of the system that decides what to do, the critic is the part that tries to predict how promising the current state is. The critic typically outputs

a value estimate parameterized by $\phi$.

Once a rollout is collected, we use rewards and bootstrap values to construct targets. If we are using GAE, we obtain advantages $\hat{A}_t$, and from those we can form return-like targets such as

Then the critic is trained to match those targets. The simplest and most common critic loss is a mean-squared error:

This part is much less mysterious than the actor. The critic is just doing regression. It looks at a state, predicts a value, compares that prediction to a target, and adjusts itself to reduce the gap. But even though the critic loss is mathematically simpler, it is essential. If the critic is poor, the advantages become poor. And if the advantages become poor, the actor starts climbing in the wrong direction.

There is still one more ingredient, and it appears for a very practical reason.

Imagine a policy in a four-action environment. Early in training, it outputs probabilities

That policy is uncertain. It is still exploring. Now imagine that after a few lucky updates it becomes

This policy is extremely confident. Maybe too confident. It may have discovered something useful, but it may also have simply latched onto a local pattern too early and stopped exploring alternatives. PPO therefore often adds an entropy bonus, also called entropy regularization.

The entropy of the policy at a state is

To understand why this term appears, it helps to step away from reinforcement learning for a moment and think about a simpler situation.

Imagine you are teaching a child to solve puzzles. On the first day, the child has no idea which strategy works best. So they try everything. Turn the pieces this way, flip them that way, test different combinations.

Their behavior is highly unpredictable.

Now imagine that after solving just one puzzle, the child suddenly decides:

“Turning the piece clockwise must always be the right move.”

From that moment on, they try only that strategy, even if it fails repeatedly. Nothing forced them to stop exploring. They simply became too confident too early. This is exactly the kind of situation reinforcement learning algorithms can fall into.

To see how this happens, consider a small environment with four possible actions:

At the very beginning of training, the policy might output

Every action is equally likely.

The agent is essentially saying:

“I don’t know which action is good yet.”

Now imagine the agent tries action $a_2$ and receives a slightly higher reward than expected. The advantage becomes positive, and the gradient update increases the probability of that action.

After a few updates, the distribution might look like this:

This is perfectly reasonable. The agent has found a promising action but still explores others.

But suppose those early rewards were just lucky noise. If the algorithm keeps reinforcing that same action again and again, the distribution might quickly become

Now something dangerous has happened.

The policy has become almost deterministic.

Action $a_1$ will be chosen nearly every time. The other actions are effectively gone from the agent’s behavior.

Even if one of those forgotten actions actually leads to a better long-term outcome, the agent may never discover it.

This is where entropy enters the picture.

Entropy is simply a way to measure how uncertain a probability distribution is.

Imagine again our four–action environment. The policy outputs probabilities

Let us compute the entropy.

For each action we multiply the probability by its log probability and then sum them. So the entropy becomes

All four terms are identical, so we can write

Since

the negative sign turns the value positive, producing a large entropy.

This is exactly what we expect. The policy is highly uncertain. It spreads probability evenly across all actions. High uncertainty means high entropy.

Now imagine the other situation we described earlier:

If we compute the entropy again,

Now something interesting happens.

The term

is very close to zero, because $\log(0.97)$ is close to zero.

Meanwhile the small probabilities contribute only tiny amounts. The total entropy becomes very small.

The first one has high entropy (+2). The agent is uncertain and explores many actions.

The second one has low entropy (+0.2). The agent is extremely confident.

A more spread-out distribution has higher entropy. A sharply peaked distribution has lower entropy. By adding entropy to the objective, PPO softly rewards the policy for staying somewhat uncertain, especially early in training.

In the language of maximization, we add a bonus:

In the language of minimizing losses, that same term appears with a minus sign:

That is why people sometimes call it an entropy bonus and sometimes entropy regularization. It is the same mechanism viewed from two different optimization conventions. The effect is to resist premature certainty. It says to the actor: “learn, but do not collapse into overconfidence too early.”

Now all the pieces are on the table.

The actor contributes the clipped surrogate loss. The critic contributes the value-regression loss. Entropy contributes an exploration bonus. So the total PPO loss used in code is often written as

Here $c_v$ controls how strongly we care about the critic fit, and $c_e$ controls how much exploratory randomness we want to preserve.

If we expand the actor term, the whole thing becomes

This equation is the point where PPO finally looks like a modern deep reinforcement learning algorithm rather than a single clever trick. The clipping gave us stable policy improvement. The critic gave us a baseline and an advantage signal. The entropy term gave us exploration pressure. Together they become one training objective.

At this point, it is also important to be honest about what PPO does not guarantee.

TRPO was built around the idea of a more explicit trust-region constraint, tied to KL divergence. PPO does not enforce that kind of exact constraint. Clipping the ratio is not the same thing as solving a constrained optimization problem. It does not mathematically guarantee that every update stays inside a precise trust region in distribution space. It does not guarantee monotonic improvement in the way one might hope from an idealized trust-region method.

So PPO has no exact trust-region guarantee.

The clipping is a heuristic. But it is an extremely effective heuristic.

There is one more reason practitioners still keep an eye on KL divergence even in PPO. The clipping usually discourages overly large updates, but it does not completely prevent them. Because of that, many implementations also monitor the KL divergence between the old and new policy during training. If the KL becomes too large, they early-stop the optimization epochs on that batch. In other words, even though PPO abandoned the heavy machinery of TRPO, many practical implementations still keep a lightweight trust-region instinct alive: watch how far the policy has moved, and stop if it drifts too much.

That detail reveals something important about PPO’s personality. PPO is not saying that KL divergence was a bad idea. It is saying that we do not necessarily need to optimize under a strict KL constraint at every step. Often it is enough to clip the probability ratio, train with minibatches, and keep one eye on KL as a safety diagnostic.

$\begin{array}{l} \textbf{Algorithm: Proximal Policy Optimization (PPO)} \\[10pt] \textbf{Input: } \pi_{\theta},\ V_{\phi},\ \gamma,\ \lambda,\ \epsilon,\ \alpha_{\pi},\ \alpha_{V},\ c_{e},\ T,\ K,\ M,\ \kappa \\[6pt] \textbf{Output: } \theta^{*},\ \phi^{*} \\[12pt] 1.\hspace{0.5cm} \text{Initialize policy parameters } \theta \text{ and value parameters } \phi \\[6pt] 2.\hspace{0.5cm} \text{Repeat until convergence:} \\[6pt] 3.\hspace{1.2cm} \theta_{\mathrm{old}} \leftarrow \theta,\quad \phi_{\mathrm{old}} \leftarrow \phi,\quad \mathcal{D} \leftarrow \varnothing \\[6pt] 4.\hspace{1.2cm} \text{for } t=0,\dots,T-1: \\[6pt] 5.\hspace{2cm} a_t \sim \pi_{\theta_{\mathrm{old}}}(\cdot \mid s_t),\quad s_{t+1},r_t,d_t \sim P(\cdot \mid s_t,a_t) \\[6pt] 6.\hspace{2cm} \mathcal{D} \leftarrow \mathcal{D} \cup \lbrace(s_t,a_t,r_t,s_{t+1},d_t,\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t),V_{\phi_{\mathrm{old}}}(s_t))\rbrace \\[10pt] 7.\hspace{1.2cm} \hat{A}_{T} \leftarrow 0 \\[6pt] 8.\hspace{1.2cm} \text{for } t=T-1,\dots,0: \\[6pt] 9.\hspace{2cm} m_t \leftarrow 1-d_t \\[6pt] 10.\hspace{2cm} \delta_t \leftarrow r_t + \gamma m_t V_{\phi_{\mathrm{old}}}(s_{t+1}) - V_{\phi_{\mathrm{old}}}(s_t) \\[6pt] 11.\hspace{2cm} \hat{A}_t \leftarrow \delta_t + \gamma \lambda m_t \hat{A}_{t+1} \\[6pt] 12.\hspace{2cm} \hat{R}_t \leftarrow \hat{A}_t + V_{\phi_{\mathrm{old}}}(s_t) \\[10pt] 13.\hspace{1.2cm} \text{for epoch } k=1,\dots,K: \\[6pt] 14.\hspace{2cm} \text{for each minibatch } \mathcal{B} \subseteq \mathcal{D} \text{ with } |\mathcal{B}|=M: \\[6pt] 15.\hspace{2.8cm} r_t(\theta) \leftarrow \dfrac{\pi_{\theta}(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},\quad t \in \mathcal{B} \\[8pt] 16.\hspace{2.8cm} L_{\pi}(\theta) \leftarrow \dfrac{1}{|\mathcal{B}|}\sum_{t \in \mathcal{B}} \min\!\Big(r_t(\theta)\hat{A}_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t\Big) \\[8pt] 17.\hspace{2.8cm} L_{V}(\phi) \leftarrow \dfrac{1}{|\mathcal{B}|}\sum_{t \in \mathcal{B}} \big(V_{\phi}(s_t)-\hat{R}_t\big)^2 \\[8pt] 18.\hspace{2.8cm} L_{H}(\theta) \leftarrow \dfrac{1}{|\mathcal{B}|}\sum_{t \in \mathcal{B}} \mathcal{H}\!\big(\pi_{\theta}(\cdot \mid s_t)\big) \\[6pt] 19.\hspace{2.8cm} \theta \leftarrow \theta + \alpha_{\pi}\nabla_{\theta}\Big(L_{\pi}(\theta) + c_{e}L_{H}(\theta)\Big) \\[6pt] 20.\hspace{2.8cm} \phi \leftarrow \phi - \alpha_{V}\nabla_{\phi}L_{V}(\phi) \\[6pt] 21.\hspace{2.8cm} \widehat{D}_{\mathrm{KL}} \leftarrow \dfrac{1}{|\mathcal{B}|}\sum_{t \in \mathcal{B}} \log \dfrac{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)}{\pi_{\theta}(a_t\mid s_t)} \\[6pt] 22.\hspace{2.8cm} \text{if } \widehat{D}_{\mathrm{KL}} > \kappa:\ \text{stop optimization (exit epoch loop)} \\[12pt] 23.\hspace{0.5cm} \text{return } \theta^{*} \leftarrow \theta,\quad \phi^{*} \leftarrow \phi \\[16pt] \\[3pt] d_t \in \lbrace 0,1 \rbrace \text{ is the terminal indicator, and } m_t = 1-d_t \text{ is the nonterminal mask.} \end{array}$

In the algorithm above, the actor and critic are written as if they were updated separately, which is the clearest way to explain PPO when the policy parameters $\theta$ and value parameters $\phi$ are independent. But in many practical implementations, the actor and critic share a common neural-network backbone, with only the final policy head and value head separated. In that case, it is more natural to train the whole network using the combined loss

and then take a single optimizer step.

The idea is simple: the shared backbone learns a common representation of the state, while the actor loss tells that representation how to become more decision-useful, the critic loss tells it how to become more value-aware, and the entropy term prevents the policy head from becoming overconfident too early.

So the algorithm shows separate updates for conceptual clarity, but when a shared backbone is used, those parts are usually merged into one total loss and optimized jointly.

Points to Note

Using data collected by the old policy, PPO optimizes a local surrogate objective weighted by the action-probability ratio. This does not make PPO fully off-policy; rather, it allows a few epochs of reuse as long as the new policy stays close to the old one.

Also, PPO does not hard-constrain the new policy’s ratio to stay within $1\pm\epsilon$; instead, it clips the surrogate objective so that pushing the ratio farther stops improving the objective for that sample.