Understanding a command and deciding whether to actually respond to it are two entirely different problems. In the past we filtered this out by bolting separate components — wake-word detection and the like — onto the front of the assistant. But today's end-to-end audio LLMs — large AI models that hear sound directly, understand it, and respond on the spot — swallow that filtering step into the same lump. A single model decides, right then and there, whether to stay silent, to answer, or to call a tool. The trouble is that we had no proper yardstick for measuring this. So the authors of this paper carved out exactly that boundary and built a diagnostic benchmark, VGBench (1,018 items), then ran six off-the-shelf audio LLMs through it. The result, boiled down to a single line: picking the right tool doesn't mean they can stop. Even the best-performing model stayed silent on only 14% of commands that came from a switched speaker.
Here, "making a tool call" means the model emits a structured call that points to a function like "add to calendar" or "send a message." That said, VGBench only checks whether that output parses into the right format and whether the named tool is correct. Whether the function actually runs, or whether the arguments (which event, to whom) are right, is left out of the scoring. It's the step past merely understanding the words — pointing at which hand to use — but whether that hand actually moves is a question this benchmark does not ask.
Voice assistants are increasingly entrusted with tasks that produce real-world outcomes. They book appointments, send messages, place orders, control devices. But "turn off the AC" could be something the user said to the assistant, something they said to the person next to them, or something they muttered to themselves. On paper, all three are identical. The difference lives in the sound and the situation: who said it, where, and in what context.
Evaluation practice misses this difference. Most evaluations pair a "well-formed request" with the "correct response" and score the match. So even when tool-selection accuracy comes out high, there's no way to tell whether that was a judgment made by reading the acoustic context or just the result of recognizing the command words.
So the authors narrowed the question. With the words of the command held exactly constant, and only the source and scene of the sound changed — does the model act, or does it hold its tongue?
VGBench splits into three settings. The crux is the third one, speaker-switch; the other two prop it up on either side.
The speaker-switch design rests on a rule laid down in advance: a wearable device's conservative permission policy. If the command comes from the wearer up close, execute it; if it came from someone else or from far away, keep quiet. You don't want the device on your wrist to open your wallet because the person across from you said "pay."
To see whether this rule holds, VGBench plays the same sentence under three conditions. In all three, the words spoken are identical.
The switch condition doesn't change speaker identity alone. It changes three cues at once: the source (A→B), the far-field acoustics, and the 600-millisecond temporal boundary. So what this benchmark measures isn't "whose voice is this" but acoustic-context gating — synthesizing multiple sound cues to open or close the gate on whether to respond. That's why the authors draw a firm line: they are "measuring multi-cue gating, not speaker identity."
There's a safeguard in the scoring as well. The speaker-switch score doesn't look only at the silence rate in the switch condition; it bundles in tool-selection accuracy from the same-speaker and text-only conditions. This is to keep a lazy policy of just staying silent at everything from scoring well.
The authors ran six off-the-shelf models (Qwen3-Omni, Nemotron, the Gemini family, Kimi-Audio, Step-Audio-R1.1, Audio Flamingo 3) along with three training-free variants that only tweak the inference procedure, all under the same conditions. Among the raw models, the best at staying silent on switched commands managed 14%; among the training-free variants, the best was 6%.
Step-Audio-R1.1 throws the gap into sharp relief. In the same-speaker condition it pinpoints the target tool at 96%, and in the text-only condition at 97%. Yet its silence rate on switched-speaker commands was 1%. It's not that it failed to understand the command. It understood perfectly and still couldn't make the judgment that "right now, I should stop." In the paper's framing: the ability to recognize a command's content and the ability to govern action from acoustic and situational evidence are two separate things.
So can this "ability to stop" be instilled through training? The authors examine this with a post-training case (post-training: the stage where a finished model is taught further) called VoxGate. The approach has two stages. First, supervised fine-tuning (SFT) — reteaching by showing labeled examples — tackles the core problem, gating, head-on. This stage alone silenced 91.3% of switched commands while still selecting the correct tool in every near-wearer command and text-only case. Stopping when it should stop and moving when it should move — both caught at once.
Then they add a GRPO stage (a reinforcement-learning family that rewards good answers) on top. Since SFT already solved the core problem, this stage is an attempt to push up the remaining performance — side-talk and self-talk accuracy and the like. The numbers rose modestly, and the authors explicitly state that this gain has not been isolated as the pure effect of the GRPO technique alone (see the supplement below for figures). It's a staged design: SFT raises the frame, GRPO handles the finishing touches.
Finally, in their factorized controls — experiments that peel off the changed cues one at a time to measure each one's effect — the authors confirmed that the source change independently affects behavior, and that the far-field acoustics alone silenced 60% of same-speaker commands. Evidence that the proximity rule really is at work.
The authors draw their own boundaries. The post-training split divided train and eval only at the item level. That is, generalization to unseen voices, unseen template families, and unseen tool groups is not demonstrated. The net effect of GRPO was not isolated either. There is as yet no estimate of human agreement for the judge model that scores free-form responses, so unlike the rule-scored silence and tool decisions, uncertainty remains in the free-response portion. And because the switch condition is designed to change three cues in a bundle, sensitivity to far-field acoustics depends on how the sound is rendered — a limit the authors point out themselves.
Picture an ONTLE AI PM watching over a client collaboration channel. Confirmed instructions get posted there, but so does "I'd love to ship this one this week," "ugh, should we just scrap that feature," and chatter traded among teammates — all mixed into the same stream. On paper, every one of them is a "sentence shaped like a command." The trap this paper showed in voice exists just the same in a text channel. Accurately grasping what a request is, and discerning whether it's an execution order handed to you right now, are different abilities — and being good at the former does not drag the latter along behind it.
So understanding a request well is no reason to relax. A separate gate has to stand there, judging "is it okay to act on this right now?" The reason ONTLE's request-intake pipeline puts an on_hold state (awaiting a decision) between every imperative sentence and the execution queue — instead of dropping it straight in — is that it builds a place to halt things into the structure. But having a place to halt and being able to accurately discern, at that place, "is this really an order given to me" are two different stories. The latter, as VoxGate shows, is a skill that has to be taught separately, and ONTLE is still in its early days with it. This is a starting point whose grain matches the direction the paper points to — not a claim that we've already built it all.
Here's where VGBench's usefulness shows. It hands us a yardstick for measuring how systematically we're cultivating this ability to "stop." If an agent's maturity is decided not by what it can execute but by what it can refrain from executing, then there has to be a way, alongside it, to measure that restraint in numbers and improve on it. ONTLE means to use this research not as grounds for self-congratulation but as a measuring tool to pull the product up.
No. It isn't about transcription accuracy. With the words held exactly constant and only the source, distance, and temporal boundary of the sound changed, it measures whether the model correctly chooses to act (make a tool call) or stay silent. It's a question of action decisions, not dictation.
You shouldn't add them that way. The 91.3% is the rate at which supervised fine-tuning (SFT) silenced switched commands, while side-talk 68.4%→70.9%, self-talk 52.0%→60.0%, and so on are figures from a different measurement that added the GRPO stage on top. They are metrics from different stages, and the authors themselves stated the GRPO gain was not isolated as the net effect of that technique alone.
The experiments assumed a wearable's permission rule (execute for the near wearer; stay silent for other sources or far-field). But the core insight — separating content recognition from action gating — carries over just as well to a text agent watching a channel, or to a system where several agents run together.