SpeakerMem-R1: AI Memory That Tracks Who Said What About Whom in Many-Speaker Conversations

This is research that makes an AI remember not just "what was said," but "who said it, who it was about, and how the situation shifted over time" in a setting where many people talk over one another. SpeakerMem-R1 stores each speaker's verbatim words and the state information distilled from them in two separate tracks, and when a question comes in, it merges the two tracks along person, event, and time axes to produce an answer. The authors report performance that surpasses existing memory systems built for conversations with many participants.

The problem this paper solves

Most AI memory systems are designed around a "one-on-one conversation." When a single user and an AI are going back and forth, you only need to store "what was said" well. There are only two speakers anyway, so there's little reason to bother distinguishing them.

But a setting where many people are talking over one another, like a group chat room or a meeting (multi-party dialogue), is an entirely different story. Simply retrieving the relevant content is not enough.

Say a project PM has been watching a team chat room for several months. They need to remember people, relationships, and timing all layered together: "this remark was A talking about B, and C had already been pulled off the task by then." Tracking, within this tangle of information, (1) who is talking about whom (speaker attribution and relationships) and (2) how each person's situation changes over time (state reconstruction) is the core challenge.

Problem (1) is about attaching each utterance to the right speaker (message attribution) while also capturing who the remark is about, how people perceive one another, and what information the group as a whole shares. Problem (2) is about gathering scattered utterances to reassemble "this person's state right now." The paper argues that this is exactly where existing general-purpose LLM memory falls apart: it loses the relationships between people and groups, and it fails to unify clues scattered across members, groups, and time.

The core idea

At the center of SpeakerMem-R1 is dual-track memory, a structure that keeps memory in two strands of different character.

Track 1 is the verbatim record, tagged with the speaker. It preserves exactly who said what, word for word. It's like keeping a meeting transcript with the speaker labels intact, so later you have grounds to settle "how exactly did that person phrase it."

Track 2 is the derived states, summarized and inferred from the verbatim record. Here the information is organized from two perspectives. Individual information is kept at the person level, and information shared by the collective at the group level. It amounts to maintaining a "profile card for each person" and a "shared team board," separate from the transcript.

When a question comes in, the two tracks aren't simply placed side by side. Instead, evidence from both sides is reconciled along three axes: entity (person), event, and time. A "question about this person" is answered along the entity axis; a "when and how did this thing turn out" question along the event and time axes, cross-checking the verbatim record against the states.

The catch is that errors creep into the process of building this structured memory. Utterances get attached to the wrong person (attribution errors), or states get updated incorrectly (update errors). And if you want a local deployment running on your own servers, you can't lean solely on a giant external model.

So the authors separately train a model that writes the memory, Writer-R1. It incorporates two mechanisms.

• SpeakerLevenshtein — a new scoring scheme that treats getting the speaker wrong as a direct penalty. Despite the edit-distance (Levenshtein) in the name, the paper notes that it does not use standard edit distance as-is. The point is that it's a strict criterion that only awards credit when "who, what, and where" were all recorded correctly. It won't let something slide just because the characters look similar, so a record that mixes up the speaker is docked immediately.
• Speaker-conditioned GRPO — GRPO is a technique that refines a model's behavior with reinforcement learning (RL); here, the training is conditioned on "who spoke." The model is trained to write memory more accurately while staying conscious of the speaker.

In short, the structure (dual tracks) decides "what to remember," and the training (Writer-R1 and its two mechanisms) raises "the ability to write that memory accurately."

Results and significance

Before looking at the numbers, let's clarify the scoring. The binary accuracy referred to here is the value obtained by grading a model-generated answer as correct or incorrect (1/0). It is not the rate of correctly answering two-choice either/or questions, so it's not a figure you can directly compare against a random guessing baseline. That's why it's not strange for the absolute scores to look somewhat low. It should be read as a signal of just how hard it is to remember multi-party conversations over the long term.

The gains this paper reports are as follows (all per the abstract).

On top of that:

• On the EverMemBench public leaderboard (per EverMind-AI's report), it scored 62.33%, which the authors state is the highest among the reported state-of-the-art frameworks.
• On all 1,986 questions of LoCoMo, it recorded 70.85%. The authors use this "two-person long-term conversation" dataset as a boundary test. It's a baseline for checking whether a system built for multi-party conversations doesn't collapse when it's actually in a one-on-one situation.
• In a controlled experiment of 305 questions, reinforcement learning lifted the average accuracy of a Writer trained only with supervised learning (SFT Writer) from 57.38% to 68.20%. This is evidence that the RL mechanisms introduced above genuinely improved the quality of memory writing.

Alongside binary accuracy, the evaluation also reports token-F1 (how much the answer overlaps with the ground truth at the token level). And in ablation studies that remove components one at a time, the authors confirmed that the verbatim track and the state track, as well as the person-level and group-level perspectives, complement one another. The point is that either one alone is insufficient, and performance comes alive only when both are present together.

In the end, this paper backs the claim that "memory for multi-party conversations isn't a problem you can solve just by retrieving well" with both structure and numbers. Keeping verbatim preservation and structured state management together in one system, and using speaker information directly as a training signal, are the core contributions.

Limitations and open questions

There are some points that, honestly, remain.

• The absolute scores are still low. The 47.9% on GroupMemBench is at once "the highest reported" and a sign that "no one has reliably solved this problem yet." It's premature to wire this memory straight into automated execution.
• The abstract alone doesn't reveal the details. The concrete criteria for splitting the tracks, the rules for updating states, the taxonomy of failure cases, and so on are not contained in the abstract. Judging real-world applicability requires checking the full text.
• Scope of generalization. Using LoCoMo as a boundary test is a good attitude, but whether the same benefit holds up in real-service conversations that differ greatly in language, domain, and group size is a question beyond the abstract.
• Cost and latency. Because it stores both verbatim and state records in duplicate and combines them along three axes on every query, both storage and computation are heavier than a single track. The authors' reason for training a separate Writer seems not unrelated to this burden. Ultimately, this research leaves the fundamental question of "how to manage the trade-off between accuracy and cost." The more you remember and the more precisely you reconcile, the higher the cost; the lighter you make it, the shakier the accuracy.

Paradoxically, these limitations tell us where and how this technology should be used.

A practical lens: so how can we use it?

ONTLE is a service that dispatches AI project managers. Since PM-style agents work by watching collaborative settings where many people come and go (Slack channels, meetings, group conversations), this paper's problem statement is effectively our problem. And the limitations above become the answer to "how should we use it."

• A design that preserves the speaker isn't optional, it's a requirement. For instance, if a PM agent gets confused about "who was it that agreed to hand over the API spec by Wednesday," it instantly loses trust. Ordinary retrieval (RAG) is strong at finding relevant sentences, but weak at preserving "who said it." So a collaboration agent must manage speaker and relationship preservation separately from retrieval accuracy.
• Holding both the verbatim and the summary lets you catch conflicts. If a state card says "C is in charge of the payment module" but a recent verbatim record says "C wants out of that task," you can cross-check the two and surface the contradiction. The dual track provides exactly this structure for "tracing back to the verbatim record when the summary is wrong."
• A design that separates the personal card from the shared team board ties directly into the status board. Keeping "progress by owner" and "shared team decisions" on different layers reduces accidents where one person's change overwrites shared team information.

That said, we have to coldly accept the fact that the absolute scores are still low. The more important the attribution or state judgment, the safer it is to have a human check it once rather than wire it straight into automated execution. At this stage, where the accuracy-cost trade-off is clear, the reasonable approach is to use this memory structure not as "a device that replaces human judgment" but as "a device that organizes things so a human can check them quickly."

References

• Paper: SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue
• Authors: Haobo Zheng, Tan Tang, Yan Chen, Weijie Wang, Yingcai Wu
• arXiv: https://arxiv.org/abs/2609.26780
• HuggingFace: https://huggingface.co/papers/2609.26780