PPO, the workhorse for training large language models (LLMs) with reinforcement learning, has carried a flaw that nobody had really paid attention to. This paper names that flaw "Value Flattening" and is the first to lay it out systematically. It then offers a strikingly simple remedy called SP³O (SParse Proximal Policy Optimization), which applies the value loss at only a few well-separated states per response (by default three points at 30%, 60%, and 90% of the response, with a fourth at 95% added for long responses exceeding 6,144 tokens). On Qwen3-Base, this consistently improved the learned policy across multiple model sizes and evaluation sets.
PPO (Proximal Policy Optimization) is the standard tool for RL on LLMs. Here, the critic plays the role of scoring "how good an outcome is this state (the point reached after generating the text so far) likely to lead to." That score is called the state value, and having it lets you reduce the wobble (variance) in policy updates and stabilize training. The critic is, in a sense, like an altimeter that tells you "how much further to the summit from here" while you're climbing a mountain.
The trouble is that this altimeter doesn't faithfully reflect the actual terrain. When the authors estimated state values using multiple Monte Carlo continuations (generating all the way to the end several times from the same point and averaging the results), they found that the true value rises and falls sharply as you pass through the intermediate states, while the critic's predictions stay relatively flat. The real terrain has steep cliffs and climbs, but the altimeter keeps pointing to a bland, gently sloping hill.
The paper calls this mismatch value flattening. To show it isn't a fluke, they reproduced the same phenomenon in FrozenLake, a simple grid environment where the state space can be controlled. In particular, they observed that the larger the state space, the worse the flattening gets. That means the LLM setting — where text is long and the number of possibilities explodes — is exactly where this problem bites hardest.
Why does the critic smear the terrain flat? The paper's analysis points to two causes.
First, an "implicit variance penalty" that hides inside the critic loss under a particular training setup. This is not a general property of squared error (MSE) itself; it arises under the specific conditions the paper considers. When the reward is given only once at the very end of a response (terminal reward) and the discount and GAE coefficients are set to γ=λ=1, standard PPO hands the same single terminal reward value as the regression target to every state (token position) within a response. If you then expand the critic loss for one response, it splits into two terms — "how far the mean of the predictions is from the target" and "how spread out the predictions are from one another (variance)" — and the latter term effectively penalizes the very fact that the predictions differ. The more densely you supervise every point, the more all predictions in a response get caught by this variance term, pushing the critic to collapse its predictions onto a single value and flatten them, regardless of what the true intermediate values are. It's like telling a test-taker to match every question not to that question's own answer but to a single number — the "total exam score": to minimize the error, they write the same prediction for every question, and the differences between questions vanish.
Second, redundant updates from temporally adjacent states. Neighboring token positions in the middle of generating text are very similar to each other. So when training the critic, these positions produce gradients pointing in nearly the same direction. Effectively the same learning signal repeats hundreds of times, drowning out the subtle real differences and pushing the predictions flat in one direction. It's like copy-pasting the nearly identical problem onto a test sheet 100 times: the student overfits to that one type and can no longer distinguish subtly different variants.
SP³O's remedy is surprisingly simple. Instead of imposing the value loss at every point of a response, impose it at only a few well-separated states. According to the paper, supervising only a few separated states (by default three points at 30%, 60%, and 90%, with a fourth at 95% for long responses) mitigates both causes at once. Pruning the redundant, near-identical signals from neighbors reduces the redundant-update problem (addressing cause 2), and the dense pressure that was flattening predictions weakens as well (mitigating cause 1). It's closer to hand-picking a few clearly distinct problems than copy-pasting 100 copies of the same one.
The paper ran experiments on Qwen3-Base. Here is what it reports.
| Item | What the paper reports |
|---|---|
| Supervision points | Value loss applied at only three points — 30%, 60%, 90% of the response (plus 95% for a fourth point when over 6,144 tokens) |
| Effect | Mitigation of value flattening |
| Policy performance | Consistent improvement across multiple model sizes and multiple evaluation sets |
Concrete performance-gain numbers or dataset sizes are not given in the abstract, so here we summarize only qualitatively. The core message is twofold. One is the diagnosis that value flattening is an important but overlooked failure mode in standard PPO critic training. The other is the prescription that a very simple sparse-supervision strategy alone can mitigate it. What stands out is that they got better results not by bolting on more complexity, but by removing learning signal.
Judging from the abstract alone, the following questions remain.
These points don't detract from the paper; they are the spots that make you want to know more about why — and how far — such a simple prescription works.
ONTLE is a "dispatched AI project manager SaaS," where our work is to make multiple agents judge and execute long work flows. The lessons of this paper land directly on that kind of product design, too.
First, "don't learn from or evaluate every moment equally." Just as the PPO critic got flattened under the redundant signal of neighboring states, an AI PM that learns from or evaluates every step of a long conversation or long work log uniformly will bury the signal from the pivotal turning points (the moments where decisions diverge). Picking out and focusing on a handful of clearly distinct decision points can lead to better judgment.
Second, "removing can be the improvement." Rather than constantly adding signals and reward terms to chase performance, pruning the learning targets sparsely — as SP³O does — can be more effective. In an agent evaluation pipeline, it's worth considering a design that concentrates supervision on well-separated checkpoints instead of scattering dense intermediate rewards.
Third, "measure the actual terrain before you trust the value estimate." Just as the paper exposed the problem by measuring how far the critic's predictions diverge from the actual Monte Carlo values, a product's success-prediction model or reward model needs the diagnostic habit of comparing against the actual outcome distribution. If the predictions are too smooth, that may be a sign not of stability but of flattening.
The longer-horizon work an agent takes on, the larger the state space grows — and, as this paper observes, the greater the flattening risk. That is a story sitting right at the center of the problem ONTLE deals with.
It refers to the mismatch where the true state value rises and falls substantially as you pass through the intermediate points of generating text, while the PPO critic's predictions stay relatively flat. It can be likened to a situation where the terrain has cliffs and climbs but the altimeter keeps pointing to bland values.
According to the paper, there are two reasons. The redundant learning signal that arose because neighboring states are similar is reduced, and the pressure that was densely pushing values toward their mean and flattening them weakens as well. Supervising only a few sufficiently separated points mitigates both effects at once. The supervision points are three by default (30%, 60%, 90% of the response), with a fourth added at 95% for long responses exceeding 6,144 tokens.
The phenomenon was reproduced in the FrozenLake environment, where the state space can be controlled, and the actual LLM experiments were run on Qwen3-Base. The abstract only states that the policy improved consistently across multiple model sizes and evaluation sets; it does not provide concrete numbers.
The more an agent handles long-running work, the larger the state space grows and the greater the flattening risk. It suggests that, rather than learning from and evaluating every moment uniformly, you should focus on a handful of distinct decision points, and keep the habit of diagnosing whether your success/reward predictions diverge from the actual outcome distribution.