When you use reinforcement learning (RL, where a model tries out its own answers and learns from the ones that go well) to make an LLM smarter, most of the time and cost goes into the rollout stage — where the LLM generates long answers on its own. That's why the idea of "just run the rollout at low precision to make it cheap" has been around for a long time, but it kept running into one wall: quality wobbles.
A team from Alibaba Token Hub and Ohio State University has now cleared that wall with TRACE. Running rollouts in FP4 (4-bit low precision), they got performance almost identical to running them in BF16 (16-bit). The core idea fits in one sentence — don't try to make each of the two paths accurate on its own; directly shrink the mismatch between the two paths.
Quick glossary - rollout: the stretch in RL training where the model repeatedly generates long answers. The most expensive part. - FP4 / BF16: the size of the container that holds a number. 4-bit (FP4) is lighter and faster than 16-bit (BF16), but it has far fewer "slots" to put values in. - MoE (Mixture of Experts): a model architecture that picks only a few "expert" modules per token.
Low-precision rollout is the idea of "quantize the model to low bits (storing values in fewer bits) only while pulling out answers, to make it fast." The problem is that FP4 has too few slots. With only a handful of places to put a value, the rollout path and the training path round the same value into different slots. When that happens, the policy (the rule the model uses to decide which token to emit next) computed by each path diverges, and as that divergence piles up, training becomes unstable or collapses.
In MoE it gets nastier. In MoE a router scores and picks experts per token, so if a tiny numerical difference jostles the score, a completely different expert fires. A single slot's worth of rounding difference ends up changing "who handles this token," and the mismatch gets amplified.
Prior FP4 RL work understood this much. But it focused on making each path close to the high-precision (BF16) original. Reducing each path's error separately doesn't mean the distance between the two paths shrinks — correcting just one path toward the original can actually widen the gap. TRACE pins exactly this spot as the root of the problem.
The name TRACE itself compresses the design. Train-Rollout Quantization Alignment via Compact GuidancE — aligning training and rollout quantization with a "small hint."
Think of it in terms of rounding. Two people round the same decimal, 2.49, to an integer. If the reference shifts even slightly, one goes to 2 and the other to 3. Both followed the rule "round to the nearest integer," yet the results split apart. This is exactly what happens at FP4's rounding boundaries. Training and rollout values that differed only minutely in BF16 fall on opposite sides of a boundary the moment they're normalized by the same scaling factor (the scale, the number you multiply by to fit values into slots), get rounded to different slots, and the difference is sharply amplified.
TRACE's prescription is simple. When the training side rounds a value near a boundary, instead of insisting on "the nearest slot," it looks at which slot the rollout actually picked and matches that. It's as if the two people at 2.49 ask each other, "which way did you round?" and go the same way. The authors show mathematically that this choice never makes the mismatch at that point worse than the default rule (the nearest slot).
For training to consult the rollout's rounding result, that information has to be recorded and handed off to the training engine. It works in theory, but in practice the sheer volume of data hits a wall fast. In the paper's Qwen3.5-35B-A3B example, even budgeting 4.5 bits per value, a single token comes out to about 50KB. Sampling 4,096 trajectories in one RL step yields up to about 51TB per step. Even assuming 300GB/s of GPU-CPU bandwidth, the transfer alone takes 3 minutes, and reading and writing at 5GB/s of storage bandwidth takes about 3 hours. An RL step has to finish much faster than that, so a design that costs 3 minutes to transfer and 3 hours to store was never even a candidate.
TRACE punched through this wall with "send only the information you actually need to decide the rounding direction." The basis is two observations. First, over 99% of the mismatched values differ by just one adjacent slot. If it's only one slot over, you don't need to send the full quantized value — sending just part of the digits that distinguish the slots (the mantissa, the part of a floating-point number corresponding to the significant figures) is enough to pin down the slot you want. Second, since this correction mostly happens in the deep layers, it only sends information from the back half of the model's layers. This greatly cuts the storage and communication burden while preserving the alignment effect.
How much faster. The rollout's throughput (the amount of tokens pulled out per unit of time) rose up to 5.4x. But there's a condition attached. It's rollout throughput compared against BF16 rollout, measured on the SGLang rollout engine alone across 4 GB200s. It does not mean total training time dropped 5.4x. What this multiple tells you is that the extra computation TRACE adds for alignment is that light.
Is quality preserved. Within the verified range, it held. Running four MoE models across three categories — reasoning, coding, and long-horizon tasks — TRACE trained with FP4 rollout matched BF16 rollout and beat prior methods under the same FP4 conditions (numbers in "Results" below).
Can we use it too. The verification was done on giant MoE models and infrastructure at the scale of 72 GB200s. Whether the same multiple and the same quality reproduce on small models and lightweight serving is a story outside this paper. What you can take regardless of scale is the design principle "shrink the distance between the two paths," and that's covered at the very end in "ONTLE's Take."
The tables have to be read together with their conditions. Every FP4 method was compared under the same conditions (weights, activations, and the KV cache — the memory that stores previously computed values to reuse — all in FP4), and the numbers are the average scores for each task.
Reasoning RL — Qwen3.5-35B-A3B. Reasoning RL is the task of solving math and coding problems. Below is the average across four benchmarks (math competitions like AIME and HMMT25, and coding like LiveCodeBench).
| Method | Average score |
|---|---|
| BF16 rollout (ceiling reference) | 74.9 |
| Vanilla QAT | 59.6 |
| QaRL | 58.6 |
| QUADS | 68.8 |
| TRACE | 75.3 |
TRACE scores 6.5 points higher on average than QUADS — the best of the prior methods shown in the paper's table — and recovers to the BF16 level. On the especially hard math benchmark HMMT25 in particular, the gap was 11.0 points: QUADS 59.0 versus TRACE 70.0. (QAT is an approach that trains the model while making it experience quantization up front.)
Larger models — coding and long-horizon tasks. DeepSWE and Terminal-Bench are benchmarks that fix real code or carry out terminal tasks; GDPval evaluates long-horizon tasks. Only each representative benchmark is pulled out.
| Model · Benchmark | BF16 | QAT | QUADS | TRACE |
|---|---|---|---|---|
| 122B-A10B · DeepSWE | 33.4 | 28.8 | 29.1 | 33.0 |
| Flash-Next · Terminal-Bench | 68.8 | 60.4 | 66.4 | 70.6 |
| 2.4T-A95B · GDPval | 90.3 | 88.7 | 85.9 | 90.2 |
In all three cases it matched BF16 rollout and beat QAT and QUADS under the same FP4 conditions. There are even cases, like Terminal-Bench, where TRACE edged slightly past BF16.
With QAT, QaRL, and QUADS, the mismatch grows as training goes on, so reward plunges late and scores fall apart. TRACE suppresses the mismatch and keeps a flow close to BF16. Here a phenomenon the authors call "low-precision adaptation" emerges — it starts out lower than BF16 due to quantization loss, but once the mismatch is kept under control, the policy gradually adapts to the FP4 environment and closes the gap. On HMMT25 it caught up to BF16 after about 120 steps and stayed comparable thereafter. The point is that the model isn't grimly enduring the low-precision environment — it's learning to fit it. That's why the authors report that a policy trained in FP4 from the start runs better in FP4 than one trained in BF16 and then lowered to FP4 after the fact.
In one line: while the other methods lose their way each trying to get closer to the original, TRACE watches only the distance meter between the two paths and makes them approach each other. The goal isn't to make each path accurate but to overlap the two forms.
The remaining limits are ones the authors draw clearly too.
The real value of this research isn't the number "5.4x" but the design principle that produced it. TRACE solved a problem specific to giant models, but in the process it surfaced a universal problem: "if the model at training time differs from the model at serving time, quality leaks." Normally we take the order "train properly (at high precision), then shrink it to be light at deployment" for granted. TRACE's result shows that order isn't always the best.
Up to here is the principle we borrow from the research. So what does ONTLE change? ONTLE handles the repetitive work of small businesses and solo founders on their behalf, and sometimes, to cut costs, we lighten and serve the model that handles customer inquiries. In that case, instead of stopping at "it's light and cheap, so it's good," our proposal is to add the following procedure before locking in a deployment.
The idea is to add one more column — "the distance between the training-time form and the serving-time form" — to a checklist that used to look only at accuracy per model, except that distance shouldn't be an abstract figure but made concrete, as above, as "the difference in pass rate between the two versions on the same inquiries."
To avoid breaking the flow of the main text, the detailed settings and method names are gathered here.
| Item | Details |
|---|---|
| Training stack | VeRL (asynchronous RL orchestration) · Megatron (training backend) · SGLang (rollout generation), all methods compared on identical versions |
| Policy optimization | GRPO (policy optimization algorithm) + R3 (replaying the rollout's expert routing at training time) |
| Stabilization | Token-level importance-ratio clipping cap 5, policy-staleness limit max_version_diff=3 |
| Quantization scope | Only MoE routed experts + the attention KV cache (the memory that stores previously computed values to reuse) are FP4 during rollout; the other modules BF16 |
| Evaluation models | Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, Qwen3.8-Flash-Next (125B total · 6B active), Qwen3.8-2.4T-A95B |
| Tasks · evaluation | Reasoning RL (35B → LiveCodeBench v6 · AIME24/25 · HMMT25, 32K responses) / Coding RL (122B → DeepSWE v1.1, Flash-Next → Terminal-Bench 2.1, 256K context) / Long-horizon task RL (2.4T → GDPval) |
| Metrics | AIME24/25 · HMMT25 averaged over 8 responses, LiveCodeBench over 10, pass@1 / Terminal-Bench · DeepSWE · GDPval averaged over 10, 4, 3 runs per task |
| Efficiency measurement | Decoding throughput = 4 GB200s (SGLang only) / End-to-end RL step = 72 GB200s |
| Training steps | 35B · Flash-Next 400 steps, 122B · 2.4T 50 steps |
What this design measures is "does running rollouts in FP4 still deliver the final performance and training stability of BF16." Since the quantization scope was limited to experts and KV, and the models are Qwen-family, this table can't speak to a full-FP4 configuration or the behavior of other families.
No. Within the verified task range it matched BF16 or slightly edged ahead in some cases, and that's a result of "low-precision adaptation" catching up on the gap. It shouldn't be read as the general claim that "low precision is superior to high precision" — the experiments are limited to Qwen-family MoE and specific tasks.
Not as-is. The 5.4x is a "rollout throughput" multiple measured on GB200, not total training time or serving cost, and there's no guarantee it reproduces on a small model. The only thing you can carry regardless of scale is the principle "check the distance between the training-time form and the serving-time form."