GameHorizon Suite, announced by Tencent ARC Lab, is an effort to build a new yardstick for "how well can an AI play games" (arXiv:2609.25001, cs.CV, 2026-09-22). Where existing game benchmarks have been boxed into one or two titles, have left out language instructions, or have leaned on a run-once, pass-or-fail counting scheme, this paper offers a standardized offline test that measures short, medium, and long horizons separately, alongside an online test that traces play back step by step. One caveat up front: as of 2026-10-03 only the paper and the project page are public — the code, benchmark, and dataset are listed as a staged release beginning around 2026-10-25, so what follows describes what the paper reports, not software you can download and run today. The walkthrough is drawn from facts stated in the paper itself; where the paper doesn't give a figure I can cite, I say so and move on.
Games make a good stage for testing AI. They demand, all in one place, the ability to read the screen, break a goal into pieces, form a plan, and translate it into precise control. And a game's goals span wildly different time scales: picking up a single item within a few seconds, winning a battle that lasts several minutes, executing a strategy that runs through an entire match. Yet all of these goals ultimately surface as the same stream of atomic actions — keystrokes and mouse movements. The paper takes this as its starting point.
The trouble is that today's AI holds this ability in two separate camps. On one side are dedicated game agents. To sustain a high control frequency, they use lightweight vision-language-action (VLA) or action-head architectures and supervise on action trajectories. Their hands are fast, but reasoning and planning are sacrificed. On the other side are general-purpose models. They plan well, but their actual control-execution ability has never been systematically measured. Some general models have been reported playing Pokémon with a separate agent harness bolted on, but because each harness is configured differently, the models can't be compared against one another.
So what's needed is a single yardstick that works for both camps — one that measures whether a model can align vision, executable actions, and natural-language goals across multiple time horizons, and does so cheaply, in a standardized, reproducible way, without being thrown off by the harness or the environment. The paper points to three ways existing data and benchmarks fall short here. First, annotation cost keeps the range of games narrow. They get boxed into the single game of Minecraft (MineDojo, STEVE-1), use only the one title Monster Hunter Wilds (WildWorld), or stay confined to simple mini-games (GameWorld). Second, language-instruction annotation is thin. Some drop instructions entirely (NitroGen, GameVerse), and some attach them only sparsely (Open-P2P). Third, online evaluation is run with only a handful of rollouts. Schemes like three attempts per scene (Lumine) or a single run per game (VideoGameBench) make comparisons unreliable, and a single success-rate number smears together distinct failure types. When a model fails, there's no way to tell whether it misread the current action, inferred the next goal wrongly, or couldn't translate the goal into the right control.
GameHorizon Suite is made of three pieces. A cooking analogy makes it quick to grasp.
First, GameHorizon-Annotator is a pipeline that attaches instructions automatically. It stacks control into a three-tier pyramid: short-horizon control (chop the onion), medium-horizon goals (make the sauce), and long-horizon strategy (get a full course out the door on time). The atomic action — the hand movement — is the same, but it gets bundled under goals of different time scales. This pipeline works from the bottom up, abstracting the finely attached instructions into higher tiers as it climbs. The key point is that, unlike the earlier approach of attaching everything by hand (Game-TARS), this one automates and scales.
Second, GameHorizon-Data is the large-scale dataset built with that pipeline. The team recruited skilled human players and, with a dedicated recording-and-upload system, captured 2K-resolution game footage together with timestamped keyboard-and-mouse control. Earlier approaches that estimated actions from web footage using an inverse dynamics model (IDM) or gamepad segmentation can diverge from the control a real person actually pressed; because this data is a genuine record of human play, it makes for a more trustworthy foundation. The paper claims it is the first public dataset to align game frames, player actions, and multi-horizon instructions together. In scale, it reports roughly 5,000 hours of recorded play across 21 games — well beyond the earlier D2E (300 hours) and gaming-500-hours (500 hours) it compares against. (Those figures are as stated in the paper and on the project page; the dataset itself is part of the staged release and is not yet downloadable as of 2026-10-03.) The games are built around AAA titles (the paper defines "AAA-centric" as a composition in which more than half of the included games are AAA) and span genres such as open-world, action RPG, competitive shooter, sandbox survival, and creature-collecting adventure.
Third, GameHorizon-Bench is the test itself. It splits in two. The offline track is closer to a standardized written exam. It bundles thousands of fixed questions into three main tasks: single-horizon action, multi-horizon instruction decomposition, and cross-horizon consistency. Added on top are fine-grained diagnostic variant tasks. The online track is a practical exam, but rather than taking it all in one sitting, it chops long-horizon play into short-horizon subtasks. It handles order-dependent causal tasks and order-free thematic tasks, verifying step by step while resetting the environment. That way, when a failure occurs, it can pinpoint exactly which step collapsed. It also serves to cross-check whether the written-exam (offline) score reflects actual playing skill (online).
On top of this suite, the paper evaluated several model families with over a million model calls. The targets were general-purpose VLMs, unified multimodal models (UMMs), coding and GUI agents, and dedicated game agents.
Three things stand out. First, a meaningful hierarchy of task difficulty emerged, and the ability gaps between models were clear. Second, a distinct positive correlation was observed between online success rate and offline score. That means cheap, repeatable offline accuracy can serve as a valid proxy for actual playing ability. Third, planning future actions and decomposing complex goals turned out to be harder than judging the current action. And feeding in medium- and long-horizon instructions, rather than vision alone, improved the planning of future actions. I don't have a verified percentage-point figure for the size of that improvement, so I'll stick to the direction. The gist is that multi-horizon instructions genuinely help.
The phrase to handle most carefully is "reproducible." It's easy to misread. The reproducibility the paper means is that, because the offline track uses fixed questions and a standardized scoring rubric, comparisons among models can be repeated with low variance. It reduces the noise of online rollouts, whose scores swing wildly with the environment and harness on every run — it is not a guarantee that running one particular model many times yields the exact same score each time. A language model's output is still probabilistic, and the paper does not claim that each run's score is identical. What standardization eliminates is "comparison noise," not "output probabilism."
The numbers, too, warrant caution — though less because they're missing than because of what they are. The paper does state GameHorizon-Data's scale (about 5,000 hours across 21 games), and it compares favorably with D2E (300 hours) and gaming-500-hours (500 hours). What I don't have a verified figure for is the size of the instruction-improvement effect, so I've kept that qualitative. And it bears repeating that the dataset, benchmark, and pipeline are still a staged release (beginning around 2026-10-25), so these numbers describe what the paper reports, not something independently re-measured from released artifacts. Also, the relationship between online and offline scores is a "positive correlation," not a perfect match. That offline can serve as a proxy for the practical exam, and that offline alone can fully stand in for the practical exam, are two different things.
The fact that the data leans on recordings of skilled human players is double-edged, too. Capturing real control makes it trustworthy, but it's expensive to collect by the same measure, and the diversity of novice or unconventional play may be relatively underrepresented. The evaluation results, likewise, should be remembered as a snapshot limited to the model families the paper chose.
ONTLE is a product that dispatches an AI PM to each client project, managing everything from intake of a request to exposure as a service. This paper's skeleton overlaps quite directly with that kind of agent product.
First, there's the finding that "planning ability and execution ability are different, and decomposing a long-horizon goal is harder than deciding the immediate action." In an AI PM, too, the hard part isn't a single tool call — it's breaking a client's blunt request into sub-goals by time horizon. This paper tells us that difficulty isn't an illusion but something real, confirmed by measurement.
Second, there's evaluation design. If you judge agent quality solely by expensive, high-variance live runs, you can't tell why a failure happened and regression status wobbles. It's better to keep two things separate: an offline regression test that repeats cheaply over a fixed question set, and a live test that resets at each stage to pinpoint which step broke. The point that a single success-rate number smears together distinct failure types is a warning that carries straight over to our own workflow reporting.
Third, there's the result that multi-horizon instructions improved the planning of future actions. It's evidence that, rather than tossing the agent only the current screen, structuring the medium goal and the higher strategy and feeding them in together raises the quality of the next action. It supports a design that explicitly hands a goal hierarchy down to the PM agent.
No. It isn't about building a new game agent; it proposes a data-and-evaluation set that measures the gameplay ability of various models in a standardized way. It's closer to building the yardstick.
No. It means that fixed questions and standardized scoring let comparisons among models be repeated with low variance — not that it eliminates the probabilism of model output. The paper, too, does not claim each run's score is identical.
The paper reports roughly 5,000 hours of recorded play across 21 games, built around AAA titles — well beyond D2E (300 hours) and gaming-500-hours (500 hours) in scale. Keep in mind these are figures the paper states; the dataset itself is part of a staged release (from around 2026-10-25) and isn't downloadable yet as of 2026-10-03.
The paper observed a distinct positive correlation between online success rate and offline score, and sees offline as a valid proxy. But since it's a "correlation," not a perfect match, offline alone can't be taken as a complete stand-in for the practical exam.