A security analyst at the next desk says: "Scan this subnet for SMB misconfigurations." A subnet is a slice of a network carved into a block of IP addresses, and SMB is the protocol Windows machines use to share files with one another. For a human analyst, that one sentence is easy. They know which tool to reach for, and their hands type the command on autopilot.
Hand the same thing to an AI and this is where it forks. "Which tool should I use?" it answers well enough. But the step of translating that tool into a single Kali Linux command that actually runs is where it stumbles, again and again. Kali is a Linux distribution that ships with more than 2,000 tools for security testing. Most of them are CLI tools. A CLI means you run a program by typing commands as text, and it cuts you no slack. One wrong letter in an option name, one dash too many or too few, and the whole command refuses to run.
So the problem splits into two distinct abilities. One is tool selection — choosing which tool to use. The other is argument construction — invoking the chosen tool correctly. Arguments are the options and values that trail behind the command. KaliBench is a benchmark that pries these two apart and scores them separately. It evaluates 24 model configurations on 8,504 query-command pairs spanning 1,642 tools. The headline finding: AI falls apart not on selection, but on construction.
Today's evaluations of "AI function calling" are fairly mature. Function calling is the mode where a model picks a predefined tool and fills in its input values. But there's a hidden premise baked in: the available tools and the format of their inputs are written out in advance as a schema. A schema is a specification that says "this tool takes these inputs in this format." With a schema, the model just fills in the blanks.
Security work doesn't look like that. There are hundreds, thousands of tools, and stuffing all those schemas into the prompt is impossible on length alone. The model has to pick a tool on its own, infer the arguments, and write a syntactically correct command from scratch — with no schema. It's less fill-in-the-blanks and more write-on-a-blank-page. KaliBench aims squarely at this "schema-free tool use." The authors note that this is the first benchmark to measure this territory this way.
It's clear why existing evaluations miss this spot. Knowledge-probing evaluations measure only what a model "knows," not its ability to move its hands. On the other end, evaluations that make a model solve a hacking challenge end to end bundle planning, execution, result interpretation, and error recovery all together. When it fails, you can't tell whether it chose the wrong tool, botched the arguments, or misread the output. KaliBench isolates just the ability to "write one command line correctly" out of that tangled lump.
A command has the shape <tool-name> <arguments>. Arguments come in two kinds. Optional arguments start with -, can appear in any order, and allow aliases. Positional arguments are bare values whose order is fixed.
Instead of tasting the finished dish, the scoring takes the cooking process apart step by step. Take the earlier "scan for SMB misconfigurations." The model's one-line answer is decomposed into three parts. Did it pick the right scanning tool (tool accuracy)? Did it attach the right option flag to turn on the SMB check (option-argument F1) — here the scoring looks at the flag name but counts synonymous aliases like -o and --output as a single flag? Did the target subnet value go in correctly (positional-argument F1) — here order is ignored, and only whether the multiset of values matches is checked? F1 here is a strict accuracy measure that reflects both what was missed and what was added wrongly. Leave something out and you lose points; throw in something superfluous and you lose points too. And "exact match" is a separate metric from these three. It awards a point only when the normalized command's argument values, and their combination, match exactly. So even perfect scores on all three construction parts don't guarantee an exact match.
Splitting it this way means that when a command is wrong, you can immediately pinpoint which of the three went wrong. And crucially, the scoring never actually runs the command. It parses the shell syntax — breaking the sentence into its grammatical units — and compares element by element against the tool name, options, and positional arguments recorded in advance for the answer. This design turns out to be the key to the training that comes later.
The core mechanism of this benchmark is a "runtime-free verifiable reward." It means you can score a command by matching it structurally against the answer, without ever running it. To make this mechanism usable, the authors went through a grueling process that threw away about 70% of the data they first created.
The starting point was Kali's official documentation (2,372 tools). A large generative model was told to produce query-command pairs grounded in that documentation — over 27,000 of them. Yet even with the docs in hand, the fabrication was severe. An initial check found that more than half of the generated commands invented facts that weren't there or carried wrong options.
So they stacked three layers of filtering. First, an AI verifier checked whether every flag actually existed in the documentation and whether the command faithfully rendered the query (more than half were dropped at this gate alone). Next, each command was run one by one in a virtual sandbox (a Docker sandbox) isolated from the real system, and anything that errored out on a nonexistent tool or a wrong argument was discarded. Finally, cases that even the strongest model failed on despite being handed the docs were inspected by humans directly — such failures are more likely a data defect than the model's fault. Lastly, pairs that overlapped in meaning were grouped and only a representative kept.
Only 8,504 made it through all these gates — 30.7% of the original pile. These were split into 5,000 for evaluation (covering all 1,642 tools) and 3,504 for training. Because the reference commands had already been filtered by sandbox execution and the scoring is mechanical, you can score the model's output and hand out rewards during training without running it each time. In effect, reinforcement learning without ever running the CLI.
The benchmark's other axis is dividing "how much tool information to provide" into three levels.
Going from U to R makes tool selection easier. Going from R to H makes argument construction easier. So by watching which interval the score jumps in, you can read off in numbers whether the model got it wrong because it couldn't pick the tool or because it couldn't assemble the arguments. The evaluation targets 24 configurations, including 14 general-purpose and 7 security-specialized.
| Metric (average) | U | R | H |
|---|---|---|---|
| Exact match | 22.3% | 28.3% | 73.1% |
| Tool accuracy | 72.0% | 95.2% | 94.6% |
| Option-argument F1 | 45.1% | 48.7% | 87.8% |
| Positional-argument F1 | 61.1% | 68.2% | 90.1% |
| Total score | 59.4% | 70.7% | 90.8% |
Read the table in two intervals and the story snaps into focus.
U to R. Narrow the candidates down to 20 and tool accuracy leaps from 72.0% to 95.2%. Picking the tool is all but solved with only a little help. Yet over the same interval, option arguments inch up only from 45.1% to 48.7%. Picking the right tool is nowhere near the same as assembling the right arguments.
The change erupts in the move from R to H. Hand over the documentation and option F1 jumps to 87.8%, exact match to 73.1%. That's more than triple the unhinted 22.3%. This is where the bottleneck reveals itself: argument construction, and within it, option arguments.
Why are option arguments uniquely hard? The model can't just find the option name in the docs. It has to work out from context that -o and --output mean the same thing, and it has to infer subtle syntactic quirks — like a tool that accepts --timeout 30 but rejects --timeout=30 — rules that aren't spelled out clearly in the documentation, picked up only from the nuance of example code. This is the problem of extracting implicit syntactic rules from unstructured documentation. It's reasoning closer to code generation than to natural-language understanding. That it only got solved once the full documentation was shown is evidence that the model doesn't carry these rules in its head.
It lands harder when you look model by model. On the strongest axis, GLM-5.2 scores 41.3% exact match unhinted, DeepSeek-V3.2 33.0%. No open model cleared 42% unhinted. Those same models pick the tool out of 20 candidates in R mode just fine — GLM-5.2 94.2%, DeepSeek-V3.2 98.3%. They can choose; they get stuck on calling correctly.
Why runtime-free rewards are useful shows up in the training. The authors picked RedSage, an 8-billion (8B) parameter security-specialized model. Then, rather than retuning the whole model, they used LoRA — a lightweight method that adjusts only a fraction of it — to add two forms of training. One is supervised fine-tuning (SFT), teaching the model to copy the answer verbatim; the other is reinforcement learning (GRPO), nudging behavior little by little toward higher scores. The score used for that nudging is exactly the runtime-free reward from before.
After the tuning, this 8-billion model closed in on the neck of DeepSeek-V3.2, a 685-billion (685B) parameter behemoth. A total score of 79.2% averaged across the three modes — up 7.5 points from the pre-tuning 71.7%, and just 1 point behind DeepSeek-V3.2's 80.2%. Despite being 85 times smaller. With the right data and mechanical rewards, a small model can get close to a far larger one.
That said, don't overstate the gain. The 7.5 points is a three-mode average, and the upside is concentrated in the hint-poor U and R. In H, where the full documentation is handed over, it doesn't necessarily beat models of its own class or larger. What the effect really amounts to is "it got better at assembling commands even without hints."
The limitations are better gathered in one place. Because the scoring leans on structural agreement with the reference command, a different valid command that achieves the same goal can be unfairly penalized. Since the evaluation centers on a single command line, multi-step work that chains commands with pipes is out of scope. The sandbox verification confirmed that a command "runs syntactically," not that it "works in the field." And since the data mixes offensive and detective tools, there's a risk of misuse.
So the question that remains: do we aim to push up the accuracy of a single command, or do we teach the operational art of chaining several commands to solve a problem? Knowing the goal and translating that goal into a precise command are different abilities, and getting one line right versus running the whole board is yet another matter. The picture of an "AI assistant that just handles everything" is further off than you'd think — and this research shows it in numbers.
This question is ours too. ONTLE attaches an AI PM to each customer project and puts it in charge from intake through execution. That PM, in the end, also does the work of translating human words into actions a machine will carry out. It's exactly the gap KaliBench takes aim at.
Let's follow one scene all the way through. A customer tells the PM: "From this week's review data, drop the duplicates and aggregate by supplier." The PM translates this into an aggregation query or calling code. Verification then splits into two layers.
First, the machine checks the specification. Before executing, the PM's query is parsed and compared mechanically against the available functions and argument formats. It's a gate that asks whether the aggregation function, the date format, and the grouping key fall within the spec. It's KaliBench's structural scoring, carried over wholesale. On top of that, rather than letting the PM compose queries in free prose, we hand it, alongside, the usage documentation for the narrow "actions" it is allowed to use. We're chasing exactly that effect where accuracy tripled once the documentation was shown.
Second, a human checks the business meaning. Even when the spec is correct, judgments like "the definition of a duplicate" or "the criterion for grouping suppliers" aren't caught by structural scoring. This layer is kept separate from the machine gate and checked on its own.
The numbers to measure here boil down to exactly two. One, split by before and after we provide the narrow-action documentation, the proportion of specification errors caught before touching the production DB. Two, over that same before-and-after, how much the PM's "words-and-actions mismatch" shrank. Let me also make one caution clear: there's no guarantee that the effect documentation showed on security commands carries over to our business queries as is. So until we actually measure those two metrics and confirm, it's only an expectation. The harder a task is to reverse — the more money and credits are on the line — the greater the worth of a pre-execution gate.
No. F1 is a measure, element by element, of how much the command structurally overlaps with the reference command. Because the scoring doesn't actually run the command, a high score only guarantees "it resembles the correct answer syntactically." Conversely, a different valid command that achieves the same goal can be unfairly penalized because its structure differs from the reference. This is the inherent limitation of structural scoring.
In theory, yes. But security tools number in the hundreds and thousands, so stuffing all those schemas into the prompt is unrealistic on length alone. That's exactly why KaliBench deliberately addresses the "schema-free" situation. There is, however, a practical compromise — instead of the full schema, providing just the documentation of the relevant tools alongside — and in the experiments that alone raised accuracy sharply.
It's too early to see it that way. The small model's gains are concentrated in the hint-poor situations (U and R), and in the situation where full documentation is provided (H) it didn't necessarily beat models of its own class or larger. Also, this is the result of specialized training on the narrow task of generating a single command line. It doesn't mean it catches up just the same on broad work that chains many steps.