Turn-taking conversation is familiar territory. You finish speaking, the model replies, then you speak again. A technical report out of Tencent's Hunyuan Speech Team, Zhejiang University, Shanghai Jiao Tong University, and others, titled "Multimodal Duplex Interaction Agent" (arXiv 2609.08977, v3, 2026-09-12), upends that ordering itself. Their system, Gander, keeps listening even while you're still talking, offers mid-stream feedback or follow-up questions when needed, and lets you cut in and interrupt its own response. Let's clear up one misconception first: Gander is not "a single clever model that handles everything from small talk to complex work." It's a system that splits apart — and then joins — a model for real-time interaction and a separate backend for deep reasoning.
One more thing worth settling before we go on. The report's initial version (v1) was titled "Omni Interaction Agent Technical Report," and the current latest release (v3) renamed it to "Multimodal Duplex Interaction Agent." Everything below follows the latest v3.
Most human-AI conversation today is built around text-based, turn-by-turn exchange: a discrete cycle where you give an instruction, the model answers, and you instruct again. Human-to-human conversation doesn't chop up that neatly. While listening, we drop in short affirmations like "mm-hm" or "and then?", we cut in before the other person is done, and we start talking or change direction based on what's happening around us. The paper argues that AI should move closer to this kind of continuous, bidirectional communication.
The authors frame natural interaction as requiring three capabilities together. First, the ability to perceive and understand information across speech, vision, and text. Second, low-latency, bidirectional, real-time communication. Third, agentic abilities such as contextual reasoning, planning, tool use, and task execution. They then pose two questions. One: how much of real-time behavior can you achieve by stringing together existing modules like VAD (voice activity detection) and ASR (automatic speech recognition), and where does it instead have to be handled as an intrinsic capability of the model itself? Two: how do you reconcile the low latency needed for real-time responses with the long-horizon reasoning needed for complex tasks, all inside a single system?
The paper's position is clear: don't hand the timing of interaction off to an external pipeline — handle it inside the model. Coordinating behaviors like interruptions, background noise, multiple people speaking at once, and short back-channel affirmations through independently designed modules is simply too prone to falling out of sync.
Gander rests on two pillars.
The first is a Cerebellum-Brain collaboration structure. The roles split exactly as the names suggest. The Cerebellum handles real-time multimodal interaction and conversational responses — the part that, in a person, reacts reflexively and keeps the flow of speech going. The Brain handles the heavy lifting: complex planning and agentic task execution. The Cerebellum keeps a constant grip on the living context of the conversation, and when deep judgment is needed, it calls the Brain via a tool call. At that point it passes along the text context accumulated so far, plus speech transcripts and visual frames. The Brain takes this, reasons more deeply in an asynchronous manner, and returns intermediate summaries, plans, or final results to the Cerebellum, which weaves them back into the ongoing conversation. Think of a café cashier and the kitchen. The cashier (Cerebellum) keeps chatting with the customer and taking orders, hands off the time-consuming cooking to the kitchen (Brain), and keeps the conversation alive with lines like "it'll be right out." While the dish is being made, the customer can keep talking to the cashier.
The key to this design is that the Brain is a plug-and-play component. The paper designs the Brain to be used inside Gander without any extra training (a training-free back brain). So when a stronger reasoning backend comes along, you can swap out just the Brain without retraining the real-time interaction model. For reproducibility, the paper explains that the Cerebellum is adapted from MiniCPM-o 4.5 as its starting point, while the Brain can be implemented with a general-purpose agent such as the Codex family or Claude Code (§3.1, §5.1). In other words, the Cerebellum and the Brain are different models, with an agent orchestration runtime bridging them.
The second pillar is the streaming Thinker-Talker structure the Cerebellum runs on. User input and model output are handled in time-aligned chunks, and these are flattened into a single ordered token stream. For each chunk, the model explicitly predicts whether it should "listen or speak" right now. By training the model to decide, moment to moment, between listening and speaking on its own, it runs perception and generation concurrently at low latency, with no separate turn-management controller.
The evaluation covered four axes: conversational ability, interaction ability, understanding, and tool-assisted task execution. Here's what the authors report. In an internal human evaluation, Gander maintained natural, expressive spoken conversation, and on benchmarks it showed effective turn-taking ability and "encouraging" results on spoken question-answering and understanding tasks. The benchmarks mentioned include Full-Duplex-Bench, SpokenQA, and WorldSense.
Here's where I'll be upfront about something. This post does not reproduce the concrete scores for each benchmark. If you're curious about the "it scored X" numbers, the accurate move is to check Section 5 (Experiments) of the paper directly. That said, the authors' strength of claim varies by where it appears. In the abstract they hold back, describing the results as "encouraging" rather than leading with "state-of-the-art," but in the body (§5.2, §5.3) they explicitly state that, within the comparison tables, turn-taking performance and full-duplex-class SpokenQA performance are the highest.
The part that carries meaning even without the numbers is the breadth of situations Gander claims to handle. It explicitly names tricky settings as in-scope: noisy environments, multi-party conversation with several people speaking at once, and short back-channel affirmations. It reportedly also shows context-sensitive behavior, such as resolving ambiguous instructions using visual context. The fact that the model and code are released, with data release planned, also carries real value for follow-up research. Note, though, that the release status depends on when you look. The official repository provides the model and code but lists the dataset as "Coming soon," and the announcement states that the release process is underway.
The paper is clear about its own limitations. The authors note that the current evaluation focuses on representative conversational situations and tool-assisted tasks, and they leave longer-horizon agentic tasks and a wider range of deployment conditions as future work. Put another way, this is less "complex workflows all work" and closer to "we've shown the promise in that direction."
The structure itself raises questions, too. Calling the Brain asynchronously via tool calls is flexible, but if the round-trips between the Cerebellum and the Brain become frequent, managing latency and consistency can get complicated. The advantage of swapping the Brain in without training, flipped around, also means the Brain's quality becomes the ceiling on the system's quality. From the excerpts alone, it's hard to judge what the real-world latency actually is, or how it behaves across languages, including Korean. I'll leave that part open rather than make a firm call.
ONTLE is a product that dispatches an AI PM to each client project, managing everything from intake to service delivery. There are a few threads in Gander's design worth pulling straight into our work.
First, the Cerebellum-Brain split is a case of structurally resolving a tension agent products commonly face: "fast response or deep processing?" You keep the front end that talks to the user light and immediate, run the slow analysis, design, and execution asynchronously behind it, and only merge the results back into the conversation. It overlaps with the flow of a PM chatting with a user while background work churns, then slipping the result naturally into the report once it's done.
The decision to keep the reasoning backend swappable without training is also worth noting. If you couple the interaction layer and the reasoning layer loosely, then when a better model arrives you can change just the back end instead of rebuilding the whole thing. Allowing interruptions and proactively asking follow-up questions also maps onto PM-style UX. Reflecting a user's change of mind immediately, and asking a clarifying question up front when something is ambiguous, beats holding on for a long time only to deliver a result that missed the mark.
That said, this isn't a call to port it wholesale. Gander is speech and real-time interaction research, and the authors themselves scoped the evaluation narrowly. What's meaningful for ONTLE isn't any particular number — it's the design principle: separate the fast layer from the deep layer, and bridge them with tool calls.
No. It's a system that combines a Cerebellum for real-time interaction (adapted from MiniCPM-o 4.5 as its starting point), a separate Brain for deep reasoning (implemented with a general-purpose agent such as the Codex family or Claude Code), and an agent orchestration runtime bridging the two. The description "a single model does it all" is inaccurate.
This post doesn't reproduce the concrete scores. The authors report natural spoken conversation in human evaluation, and in the abstract they describe the understanding results modestly as "encouraging," but in the body (§5.2, §5.3) they explicitly state that, within the comparison tables, turn-taking and full-duplex-class SpokenQA performance are the highest. For the exact numbers, see Section 5 of the paper.
The model and code are released, and the data is planned for release (per the official repository, the dataset is "Coming soon," with the release process underway). The authors themselves note that the evaluation is limited to representative conversational and tool-assisted situations, and that longer-horizon agentic tasks and a variety of deployment conditions are future work. Before productizing, you'd need to separately validate real-world latency, language support, and so on.
The Brain is designed as a plugin that needs no additional training inside Gander. So when a stronger reasoning model comes along, you can replace just the Brain without retraining the real-time interaction model (the Cerebellum). The trade-off: the Brain's quality sets the ceiling on the system's quality.