It's tempting to assume the more of someone's history an AI reads, the better it knows them. In practice, the opposite keeps getting reported. Feed a model a longer behavior trail and its next-action predictions often get worse, and a model with stronger general ability can actually do worse at serving one specific person. The logs pile up, yet the model still can't call that person's next move.
Vela is a framework aimed squarely at this problem. The idea is simple: what a user "does next" is already sitting in the data as the answer. Cut the history at a point, show the model only the earlier part, and have it predict the behavior that actually followed. Nobody has to hand-label anything — what the person actually did becomes the answer key. When the authors turned anonymized user behavior logs into these "question-answer" pairs and fine-tuned a model on them, not only did user understanding improve, so did general abilities like math and science — and this piece also weighs how far those numbers should be trusted.
When an LLM moves into personal assistants, recommendation, or advertising, the counterparty changes. It's no longer an average person but this one specific person (individuality); it has to reason over behavior spanning weeks to years (long horizons); and every judgment has to rest on that person's actual record (grounded in evidence). The paper calls the ability where these three converge the "user dimension of LLM intelligence" — the power to read one person's preferences, intentions, and future actions out of their past.
LLMs are weak here today. Most of them learn from web pages, textbooks, code, and machine-generated synthetic data, and almost none of that captures how one individual's behavior shifts over time. The result is the paradox above: longer histories lower accuracy, and the model gets dragged toward the "average user," failing at personalization.
What the authors zeroed in on is the real behavior logs already accumulated at massive scale inside content platforms and personal assistants. Human-written public text is running dry, while behavior data keeps growing. The remaining puzzle is turning these ragged logs into "a diverse set of problems with clear answers."
There have been earlier attempts to work with users, but they were either locked to one specific application like recommendation or summarization (P5, RLPF, User-LLM), or stopped at evaluating how well a model remembers and predicts users (LongMemEval and others). Where Vela parts ways is that instead of one narrow application, it stamps out a broad set spanning 91 task types under the same "behavior = answer" principle, and asks whether the effect spills past user tasks into general ability too.
Vela's idea resembles how you'd write an exam question. Take a person's chronological record and cut it at a point. The earlier part is the "visible history" shown to the model; the later part is the "future" you hide. The model has to look only at the front and predict what actually happened in the back. There's no need to invent answers — what the person actually did is the answer.
Records fall into six categories: views, searches, purchases, posts, follows, and the negative feedback revealed by hiding a post. Identifying information like names, contacts, and accounts, along with profiles, is stripped before questions are built, and any record still carrying identifiers is dropped wholesale.
The questions split into two broad kinds: predictive ones, scored directly against the hidden future, and rubric-based ones, which don't resolve to a single answer and so are evaluated with a per-task rubric. Take one user as an example. Say that over the past two months they bought running shoes, athletic socks, and a sweat-wicking shirt in sequence — this single record spawns four questions.
The attribution question has already revealed the future behavior — "they bought a running jacket." So if you also attached a predictive question to the same behavior, the predictive answer would just leak. That's why each future behavior gets only one of the two — predictive or attribution — assigned at random, so you're never handing over the cheat sheet alongside the question.
Tasks are built in four steps. For each step, the "why" matters more than the "what."
The detailed settings are consolidated in the table below.
| Item | Setting |
|---|---|
| Split length | 72K–120K tokens (Qwen3.6-35B-A3B tokenizer), individual behavior records never split |
| Visible history : future | front ~2/3 : rest |
| Multiple-choice distractors | drawn from a pool of other users |
| Question/rubric writer, grader | Qwen3.8-2.4T-A95B, highest reasoning level (xhigh) |
| Difficulty measurement | base model responds 4 times per problem with thinking off → average score estimates difficulty |
| Trace generation/selection | 4 generated per problem → top 1, adopted only if it exceeds the difficulty average |
| Final scale | 100K users → 100K candidate tasks → Vela-Instruct ~75K (1 trace/task), Vela-RFT 9,375 |
The two corpora that branch off in the table are the crux. The traces for Vela-Instruct are made by the much larger Qwen3.8-2.4T-A95B, while the traces for Vela-RFT are generated by the base model (Qwen3.6-35B-A3B) on its own. In other words, one side has "distillation" mixed in — copying a bigger model's answers to learn — while the other uses only its own behavior-grounded solutions. This difference is decisive when you read the results.
Evaluation comes in three bundles.
Let's nail down what this design does and doesn't measure. VelaBench is performance within the format Vela created, so it has a "grading its own homework" quality. The model that wrote the problems and the model that grades them are the same Qwen3.8-2.4T-A95B, leaving room for the grader's taste to color the score. That's why the four independently built user OOD benchmarks have to be there alongside it, to show this "isn't just overfitting to the Vela format." Conversely, these benchmarks are all behavior-log based, so they measure "user understanding" well but say nothing about the outside — things like real-time conversation quality or safety.
Pulling the key scores into one table:
| Benchmark | Base | Vela-Instruct (75K) | Vela-RFT (self-generated, 9,375) |
|---|---|---|---|
| VelaBench (in-domain) | 51.5 | 63.0 | 53.5 |
| User OOD average | 41.3 | 47.5 | +1.6 vs. base |
| General OOD average | 54.2 | 58.2 | 54.2 |
Vela-35B-A3B, trained with Vela-Instruct, climbed 11.5 points on VelaBench (51.5→63.0), beating the far larger DeepSeek-V4-Flash (284B) and Qwen3.8-2.4T-A95B (2.4T) on this benchmark with just 35B parameters.
The biggest gains and the non-gains split apart. On user OOD, ad recommendation jumped 11.2 points (51.1→62.3) and profile summarization 10.1 points (20.4→30.5), while purchase intent sat essentially flat at 43.8→44.0. Not every user task improves equally.
What's intriguing is that it learned only from user data, yet seemingly unrelated general abilities rose along with it. The general OOD average went up 4.0 points (54.2→58.2), with instruction following up 6.7, in-context learning (CL-bench Life) up 5.6, and science (GPQA-Diamond) up 4.6. The authors read this as evidence that behavior supervision broadens the foundations of general intelligence itself.
Now the question is how far to trust the numbers. Vela-RFT, which used only its own solutions, gained 2.0 points on VelaBench (51.5→53.5) and stayed level with the base at 54.2 on general OOD. The gap between the headline 11.5 points and this 2.0 matters. But the two conditions differ by more than just "distillation or not." Vela-Instruct trains on roughly 75K tasks, Vela-RFT on 9,375 — the training scale itself differs. So you can't cleanly carve out the distillation share from the 11.5 points, nor declare "the effect of pure behavior supervision is 2.0 points." What can be said is two things: the headline number has a share handed down from the large teacher model mixed into it, and even with no distillation, using only its own solutions (RFT) raised user ability without eroding general ability. The authors cite this second fact as grounds that you can sidestep the known risk that "repeatedly training on model-generated data collapses the original distribution" by tying it to actually observed behavior. That said, the paper only introduces this risk as discussion — it doesn't empirically demonstrate that model collapse is actually prevented under repeated, recursive training. One round of RFT holding the general-score average steady can't by itself establish collapse avoidance and its cause — this part remains closer to an unverified expectation.
Push one step further, and there's a boundary built into the very premise that "the behavior someone actually took is the answer." Behavior is a reliable proxy for preference, but not the same thing. A click you made only because a recommendation shoved it at you, an impulse buy driven by a discount, a revisit made for lack of an alternative — these are each "a behavior" but not "what was wanted." What Vela predicts is strictly "the next action," and that doesn't always overlap with "what's wanted next." So this approach is strong at behavior prediction but walks into problems carrying this limit whenever it has to read what a user never put into words.
Across scaling experiments too, the direction held consistent. VelaBench jumped from 51.5 to 59.2 with just 4.7K samples and reached 63.0 at 75K, and since it kept rising even over the 37.5K→75K stretch, there's still room to scale further. The user OOD average was 43.5 at 4.7K and 47.5 at 75K, and general OOD was 55.8 at 4.7K and 58.2 at 75K — every scale beat the base.
Finally, the authors stacked reinforcement learning (additional training that rewards better answers to push the model further, GSPO) on top of the foundation laid by behavior supervision. RL starting from the base hits 58.5 on VelaBench, but starting from Vela (the 18.8K checkpoint) hits 66.2; profile summarization was 29.9 vs. 37.6 — both gaps 7.7 points. On top of that, the base-started version fell short even of its own pre-RL checkpoint score. The point is that a floor laid by behavior supervision also supports the RL built above it better. This 7.7 points is a different experiment from the earlier 11.5, so don't read them together.
Since we've seen the observed changes above, here we only flag the constraints that bind when interpreting those numbers.
Picture ONTLE's AI PM supporting one small-business owner for weeks on end. This owner opens only the inventory report every Monday and skips the marketing proposals each time, always replies same-day to mail from one particular vendor, and approves tax invoices only right before the deadline. Right now the AI PM can remember this pattern but can't turn it into prediction — at least, that's a region we haven't verified yet.
Vela's framework throws out a hypothesis to test here: if you set the owner's past actions as the "visible history" and the decisions they actually made this week as the "answer," then training problems emerge from company data without anyone labeling them. Rather than leave that as a vague claim, let's narrow to one task and write down the verification design.
One product judgment is needed here. Just because this owner always skips the marketing proposals, is it really good support for the AI PM to keep quietly tucking them away? Behavior prediction would answer "they'll skip it," but a good aide is the kind that sometimes pushes back with "this one you should look at." A design that treats behavior as the answer follows a user's past well, but it also imports the risk of making them repeat that past. If ONTLE adopts this approach, where to draw the line between "calling it right" and "helpful meddling" is the real design decision.
Scale, though, is the real-world wall. The paper's effect came at the 100K-user level, and a single small business's few weeks of logs don't reach that scale. There are two realistic paths. Pool anonymized behavior from many client companies under common categories (view, search, approve, reject — in a PM context) to grow the cross-signal, or, like RFT, start by verifying a conservative fine-tune that "adds only user adaptation without eroding general ability" from a small pool of self-generated solutions. The fact that the scaling experiment showed gains starting at the 4.7K scale is justification to start small without a full retrain.