In one controlled experiment, LLMs were asked to imitate a rock-paper-scissors opponent. They nailed how often the opponent threw rock, paper, and scissors. But when it came to figuring out the sequential rule — the one that decides the next move based on what was just played — and following it all the way through, they got it wrong, over and over. The breakdown was worst for rules that require looking two steps back.
Why does this matter? Because LLMs are increasingly being used as simulators that "behave like people" — virtual users reacting to recommendations, counterparts in multi-turn conversations, and the like. If a simulator matches the surface proportions but misses the rule that produced them, every decision built on top of it goes sideways too. Imitating surface statistics and seeing through to the rule that generated those statistics turn out to be very different abilities — more different than you'd think.
Two dishes made from the same ingredients in the same proportions can be produced in completely different ways. One might have followed an order — "after salt, add water" — while the other threw things in with no order at all. Looking at the finished plate alone, you can't tell them apart. Observed behavior doesn't pin down the process that produced it — that's where the paper begins.
The same goes for the behavior an LLM imitates. Even if the surface proportions of ingredients are identical — how often each move was thrown — the order of adding them can differ. This "cooking order" is the sequential rule. A rule that looks only one step back is a first-order dependency: like "water after salt," it decides the next move from the immediately preceding one. A rule that reaches two steps back is a second-order dependency: something like "if you've added flour and eggs, add sugar this time."
So the paper treats behavior as a recipe rather than a finished dish, and grades three things separately. Are the ingredient proportions right (frequency)? Did the model recall the correct recipe (strategy recognition)? And did it cook all the way through according to that recipe (rule execution)? The key point is that getting the first right guarantees nothing about the second and third. Even with identical ingredient proportions, one cook may simply have drawn at random from the same distribution each time, while another followed an order.
To grade, you have to know the answer. The researchers need to hold, from the start, the rule the opponent actually followed — only then can they tell whether the model's answer was right or wrong. So they chose rock-paper-scissors as a controlled stage. It's a minimal game where you pick just one of three options each round, so the opponent's responses lean naturally on the past, yet the set of possible moves is small enough to control and grade all the way down to the rule.
Two kinds of opponents took the stage. One is a statistical player that never looks at the past at all — always throwing the same move, or drawing independently from a fixed distribution each round. Here the question is only whether the model recovers the overall proportions of behavior. The other is an opponent that decides its next move from the last few moves — a Markov player. It varies the length of history it conditions on: the single preceding move, the pair of moves the two players last threw, or two moves back. This side tests whether the model can infer and follow sequential rules beyond mere frequency.
Finally, they added a single-player "continuation" task that strips away the meaning of rock-paper-scissors entirely, while stretching the length of history to track from 1 up to 8. It's a device to confirm the earlier findings weren't an artifact of "because it's rock-paper-scissors." The experiment spanned everything from the latest large models to lightweight ones. The core models were the DeepSeek-V4-Flash family (deepseek-chat and deepseek-reasoner) plus GPT-5 and GPT-5-mini, with GPT-4.1, Gemini 3 Flash, and Qwen3-8B added for strategy identification. Providers, sizes, and open-vs-closed status were all mixed, and every model got the exact same task instructions.
Taking the three grades from easiest to hardest, scores drop as you move toward the hard end. The easiest — frequency, recovering the opponent's overall proportions of rock, paper, and scissors — the models handled with ease. Pass. The trouble was the other two.
The second criterion, strategy recognition, is where the first weakness shows up. Across all seven model configurations, identification accuracy for history-conditioning Markov strategies was lower than for strategies that don't condition on the past. The gap was statistically significant in six of the seven, with GPT-5 the only exception.
Here's the sharpest contrast. Feed the same records to a classical statistical model instead of an LLM, and it got them 100% right at every history length (Appendix E). That means the evidence needed to tell the answer apart is already there, in abundance, inside the trajectory. So when the LLM failed, it wasn't for lack of information — it was a failure to pull that evidence out and use it. In fact, when the past records were stretched from 100 rounds to 1,000, identification didn't improve — it got worse. The bottleneck wasn't the amount of evidence, but the ability to grasp the structure inside it.
Move to the third criterion, rule execution, and a bigger problem surfaces. The linkage comes first. Getting the opponent's strategy right improved rule compliance — especially whether the model never once broke the rule across the whole evaluation window (strict rule match). Even when it got the strategy wrong, it wasn't completely off: across several models, the share of moves that happened to coincide with the rule ran above chance (about 33%, one in three).
But a bigger warning comes attached. Even when the model recognized the opponent exactly, it broke down more badly when it came to actually carrying out that rule. In the distribution-imitation task, smaller error against the target distribution is better (this error is called MSE), and the error when imitating a Markov opponent was, on average, 8–9% larger than when imitating a statistical opponent. So far that doesn't look like much. But narrow down to just the cases where the model nailed the opponent's strategy, and the gap blows past 100%. The error more than doubled. A signal that the model knew who the opponent was and still broke down on executing the sequential rule.
There was a reverse direction too — cases where the strategy was wrong but the surface behavior looked plausible. A good share of the incorrectly recovered strategies were close to the target frequencies (41.8% of 282). In other words, matching surface frequency often masked a wrong sequential rule. Both directions point the same way: recognizing the right rule and executing it are related but not the same ability, and getting one right doesn't guarantee the other.
Which kind of structure makes rule execution hardest comes through clearly in an experiment that zoomed in on a single model (deepseek-reasoner). Adding the pair of moves the two players last threw (the joint state) to the conditioning wasn't much of an obstacle — joint-state conditioning held up fine, whether first- or second-order. Where this model collapsed was on rules that require remembering two steps back: second-order dependencies.
Both second-order settings had lower strict rule match than the first-order ones. The share of moves coinciding with the rule, however, stayed relatively high. The reading is that the model recovers fragments of the rule but slips when assembling those fragments into a single coherent second-order rule. And throughout, strategy identification stayed high — so even for this model, the hard part was executing, not recognizing.
So is this failure an inability to understand and apply the rule right in front of it, or an inability to sustain it across a long stretch? The diagnostic that separates the two is "teacher forcing." In plain terms: show the model the correct record from start to finish, then have it predict just the very next move. It's like letting the model work a single problem with the answer sheet open beside it. If it still gets this wrong, the problem is the ability to apply the rule in front of it (an application failure); if it does this well but wobbles only when generating a long sequence on its own, it knows the rule but can't carry it through to the end (a maintenance failure). On a task where chance is about 33%, here's how it came out.
| Model | Overall | Among strategy-correct cases only |
|---|---|---|
| DeepSeek Chat | 52.5% | 66.7% |
| DeepSeek Reasoner | 93.3% | 97.2% |
| GPT-5 | 82.5% | 89.5% |
| GPT-5-mini | 54.2% | 70.0% |
The right-hand column keeps only the cases where the model correctly identified the opponent's strategy. DeepSeek Reasoner and GPT-5 did well when asked to predict just one move. Their free-generation errors point to "can't sustain across a long stretch" rather than "doesn't know the rule." DeepSeek Chat and GPT-5-mini, by contrast, wobbled even when asked to predict a single move, so the bottleneck is closer to applying the rule in front of them. Scaling the model up didn't make the weakness disappear so much as move it from "application" to "maintenance."
Because rule compliance is measured on top of the record the model generates itself, an early mistake propagates right through to the end. Quality was therefore all but decided in the opening stretch of generation. Strong models dipped slightly early on and then stabilized; weak ones were mostly flat and low from the very start.
The picture was the same in the single-player continuation task that strips rock-paper-scissors of all meaning. As the history to track grew longer (higher-order dependencies of second order and beyond), both context use and rule recovery collapsed together. Decisively, an estimator that just guesses from simple frequencies over the opening was more stable than the LLM's free generation in this higher-order regime. A plain, classical approach that neatly writes down the state beat the LLMs in exactly that regime.
The paper's question isn't "did it win?" but "did it win the right way?" That makes the results hard to carry over wholesale to every LLM task, and the authors draw clear lines themselves. Because it's a closed environment with strategies fixed in advance, the causality is clean — but real human behavior is messier and more open-ended. There's also room for results to shift depending on how the trajectory is presented (Appendix H). Rock-paper-scissors and the single-player task are deliberately minimized settings, so extending them to real tasks like tool use or long conversations is "suggestive," not "proven."
That said, the fact that the same higher-order dependency failure appears in the single-player task with the meaning of rock-paper-scissors removed supports the idea that this isn't a rock-paper-scissors-only phenomenon. And the experiment that dug deep into second-order dependencies is a single model's result, so the claim "second order is the wall" holds most strongly for that model. Extending it across many models remains homework.
ONTLE dispatches an AI project manager to each client project. There's one judgment call that manager faces every day: whether to automatically advance work that's "waiting on a decision" to the next step when there's no response from the client (ONTLE has a setting that proceeds automatically after a set period of silence).
And there's a tempting shortcut here: "This client always approves, so let's go ahead instead of waiting." It's easy to pull the average "high approval rate" from past records and recommend proceeding. The problem is what comes next. What if a particular client's real decision rule is conditional on the immediately preceding interaction — something like "approve if shown the cost breakdown first, otherwise hold"? The average approval rate may be high, but skip that condition on this item and the outcome flips to hold.
What this paper gave us isn't an answer but a name. It put a sharp name on the very distinction we saw earlier — that the ability to match average frequency and the ability to execute a conditional rule are different, and the latter is weaker — and in doing so, it brought into focus a risk we'd only dimly been wary of.
So on this kind of item, ONTLE moves like this. The client's high approval rate is treated as reference only, never as grounds for auto-advancing. "Usually approves" doesn't guarantee "approves this one too." Instead, we separately check whether this item actually meets that client's approval conditions — was the cost breakdown shown first? And only when we can answer "yes" to that check does auto-advance execute; the ruling is held by code, not by a human's eyeballing. The more irreversible the auto-advance, payment, or deployment, the more this holds.
The AI PM proposes "what to do," while the conditional ruling on "when to auto-advance" is held by code. No matter how plausible the approval-rate prediction, we separately confirm whether this item meets the approval conditions. That one line keeps premature automation from misreading a client's hidden decision rule and breaking trust.
The paper includes exactly that condition. Section 5.2 sets up a configuration that hands the model the rules and transition probabilities directly, and Section 5.3 along with Appendices U and W report that even spoon-feeding the rules this way did not consistently resolve the failures on higher-order dependencies. In the rock-paper-scissors experiment, too, the rules of the candidate strategies were explained, and the model was made to infer which of them the actual opponent was using. So "what happens if you write the rule out plainly" is a question this paper actually examined, and the answer is closer to "giving it more information doesn't make the problem solve itself."
This is a diagnostic study that measures "where and how it slips," not "how to fix it." Every model got the exact same task instructions, and no particular model was separately trained to fit this task. So whether "training solves it" can't be said from these results alone. All that's certain right now is where the weakness lies; how to close that weakness is a separate question.
That 100% comes from a controlled setting where the candidate rules are closed and the state can be tracked neatly. Real work isn't that clean. So the point isn't "ditch the LLM" but to divide the roles. Put an explicit, state-tracking estimator at the junctions where higher-order dependencies are central, and use the LLM where that estimator is weak.