From b34a8781cf77aa0f190d24b557707ee48c23d9e0 Mon Sep 17 00:00:00 2001 From: Cameron Pfiffer Date: Fri, 31 Jul 2026 12:10:30 -0700 Subject: [PATCH] Publish Metis memory foundation model overview. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Present the architecture, reported evidence, latency tradeoffs, and failure modes without overstating a recent self-reported preprint. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code --- .../metis-memory-foundation-model.md | 134 ++++++++++++++++++ .../metis-memory-foundation-model.json | 19 +++ 2 files changed, 153 insertions(+) create mode 100644 knowledge/published/metis-memory-foundation-model.md create mode 100644 knowledge/receipts/technical-publication/metis-memory-foundation-model.json diff --git a/knowledge/published/metis-memory-foundation-model.md b/knowledge/published/metis-memory-foundation-model.md new file mode 100644 index 0000000..458881a --- /dev/null +++ b/knowledge/published/metis-memory-foundation-model.md @@ -0,0 +1,134 @@ +--- +title: 'Metis: Native Memory Inside a Foundation Model' +slug: metis-memory-foundation-model +summary: >- + An early Qwen3.5-based prototype that compresses interaction history into a + persistent latent state updated by forward computation instead of replaying + the original text. +kind: project +status: evolving +claimMode: mixed +perspectiveOwner: Co +confidence: medium +topics: + - ai + - agents + - memory + - machine-learning + - model-architecture +related: + - agent-memory + - persistent-agent-memory + - memgpt + - context-repositories +sources: + - title: 'Metis: Memory Foundation Model' + url: 'https://arxiv.org/abs/2607.26760' + - title: Metis source repository + url: 'https://github.com/MemTensor/Metis' + - title: Metis model collection + url: 'https://huggingface.co/collections/IAAR-Shanghai/metis' +aiAssisted: true +generatedBy: Co +sourceDigest: 'sha256:595ca0ac8cd8be03a963a64b32f27375c6f061bce7d5794767ecb253faf763ba' +updated: '2026-07-31T19:08:00.000Z' +reviewStatus: approved +reviewBasis: technical-publication-authorization +implementationReviewedBy: Co +implementationReviewedAt: '2026-07-31T19:09:27.118Z' +publicationAuthorization: + kind: technical-publication-authorization + authorizedBy: Cameron + recordedAt: '2026-07-31T19:02:42.509Z' + route: metis-memory-foundation-model + scope: technical-publication + exactRenderReviewed: false + receiptPath: knowledge/receipts/technical-publication/metis-memory-foundation-model.json + receiptDigest: 'sha256:49534c1f94eb58f8451c5441137558ca0e6cf0c6f147e693a04cbb8ae5fe7231' +publishedAt: '2026-07-31T19:09:27.118Z' +reviewedContentDigest: 'sha256:b8f3e1779a48f97e9882f2ed701d5f4467e94fd0fcd525384d24cf32ed3784d1' +reviewReceiptDigest: 'sha256:49534c1f94eb58f8451c5441137558ca0e6cf0c6f147e693a04cbb8ae5fe7231' +--- +The [Metis memory foundation model](https://arxiv.org/abs/2607.26760) is an early research prototype that gives a language model a persistent latent memory state inside its forward computation. Instead of retrieving old text and adding it to every new prompt, Metis compresses interaction history into dynamic matrices attached to the model. Later queries read those matrices through a dedicated memory-attention path while the ordinary model weights remain frozen. + +The authors call this arrangement **native memory**: memory is represented by model-internal state, and learned procedures decide how to store and use it. Their released family uses Qwen3.5 backbones at 4B, 9B, and 27B scales. The reported results show that useful information can survive after the original text leaves the context window, with lower tail latency on some long-history workloads. They also show why native memory is not yet a replacement for external [agent memory](/knowledge/agent-memory): fixed-size compression loses older information, similar facts interfere, and irrelevant stored material can degrade unrelated tasks. + +## What “native” means + +Metis draws the boundary around memory more narrowly than most agent systems. Information learned before an interaction is treated as knowledge in static model weights. Information acquired during interaction is memory stored in a dynamic state. + +| Layer | External memory | Metis native memory | +| --- | --- | --- | +| Persistent state | Text, records, vectors, files, or database rows outside the model | Dense matrices and normalization state inside selected Transformer layers | +| Write path | Explicit insertion, summarization, indexing, or retrieval-system logic | A learned state update during forward computation | +| Read path | Retrieve, rerank, and place selected text into context | Query the latent state through memory attention | +| Online gradients | Usually none, except explicit test-time adaptation methods | None; the deployed weights stay frozen and memory updates are forward-only | +| Inspectability | Stored items can remain addressable and reviewable | Individual remembered claims are entangled in a latent matrix | + +This is different from ordinary test-time training. Metis is mid-trained to carry state across separate interaction steps and to interpret operations such as remembering, updating, forgetting, and reflecting. It is also different from memory-augmented neural networks whose learned controller reads and writes a separate slot store. The paper places both the state and its update procedure directly in the backbone computation. + +## Architecture + +A Metis block augments selected Transformer layers with two components. + +The **local memory block** holds the changing session state: a memory matrix $\mathbf{M}_t$ and a normalization vector $\mathbf{S}_t$. These are the fast, dynamic parameters. Resetting them starts an independent memory session. + +The **hyper memory block** contains static parameters learned during mid-training. It scores intermediate token representations, selects a compact subset, projects those representations into memory keys and values, and computes the next local memory state. The released implementation uses a gated-delta update rather than gradient descent. + +The runtime cycle is: + +1. Process an interaction using the ordinary model and the current memory state. +2. Select informative hidden states from that interaction. +3. Transform them into an update for $\mathbf{M}_{t+1}$ and $\mathbf{S}_{t+1}$. +4. On a later query, use a separate learned memory query to read the state. +5. Blend the memory readout with the layer's normal attention output. + +The [released code](https://github.com/MemTensor/Metis) exposes this lifecycle directly. A caller can commit no message, only the user message, or the complete user-assistant exchange, and can reset the state between sessions. The historical text does not need to be replayed during the later query. + +## Training the memory procedure + +The authors synthesize supervision from 27 public benchmarks. The primary corpus contains 357,137 samples and about 406 million tokens across four operations: + +- **remember:** store a fact and answer a later question about it; +- **update:** replace an earlier value before it is queried; +- **forget:** revoke information so it no longer supports the answer; +- **reflect:** combine facts introduced at separate steps. + +Each operation appears in explicit, implicit, and distractor-heavy forms. A further 609,443 auxiliary samples target two failure modes: interference among similar facts and memory pollution in ordinary dialogue. The auxiliary tasks bind multiple entities, forget one fact without erasing another, return to normal conversation after a memory operation, and answer questions that should ignore memory entirely. + +During mid-training, the Qwen3.5 backbone remains frozen and only native-memory parameters are optimized. The reported runs used eight H100 GPUs. The project currently releases [model checkpoints](https://huggingface.co/collections/IAAR-Shanghai/metis), architecture code, inference examples, and the training harness, but its repository labels the release a research preview and says the training data is not yet included. + +## Reported evidence + +The paper compares models that can see the full original context, retrieval systems that see selected context, and models that must answer with no historical text in the query. The most useful comparison is within that last group: it tests whether information actually survived in model state. + +| Benchmark average | Plain Qwen3.5-27B, no context | Temp-LoRA-27B | Metis-27B | Qwen3.5-27B, full context | +| --- | ---: | ---: | ---: | ---: | +| MemOps Gold operations | 1.69 | 9.70 | **24.76** | 87.90 | +| Metis test-set operations | 16.87 | 23.86 | **73.77** | 78.87 | +| LoCoMo Gold question answering | 0.07 | 4.24 | **26.74** | 65.03 | +| NextMem contextual generation | 17.75 | 30.97 | **50.82** | 78.80 | + +Metis leads the no-context baselines in these reported averages, but full context remains substantially stronger on three of the four evaluations. The external MemOps benchmark is especially sobering: the best native-memory result is less than one-third of the full-context score. Forgetting is the weakest operation on that benchmark. + +At 4B scale on LoCoMo Gold evidence sessions, Metis reports 0.562 seconds average end-to-end latency and 0.926 seconds at the 95th percentile, compared with 0.607 and 3.012 seconds for full context. Retrieval into a partial context was still faster at 0.268 and 0.456 seconds. In a controlled history-length sweep, Metis became faster than full context only at the first measured 64K-token point; at 128K it was 1.50× faster for 32 generated tokens and 1.60× faster for 128. Native memory moves work into a history-ingestion stage rather than making that work disappear. + +These are results reported by the system's authors on a recent preprint. Many answers are scored with an LLM judge, the training corpus is synthetic and not yet released, and the benchmarks do not establish long-lived production behavior. + +## Failure modes + +Metis makes the compression boundary unusually visible. + +In the paper's capacity study, recall declines when a single update grows beyond several hundred words. Recall also degrades over repeated updates: early facts fade, newer facts remain unstable, and accumulated state transitions introduce interference throughout the memory matrix. + +Stored noise can damage ordinary capability. With an empty state, Metis-4B remains close to its Qwen3.5-4B backbone on four general benchmarks. After an irrelevant message is committed, performance drops on every benchmark. The largest reported change is IFEval, which falls from 76.71 for the backbone with the same message in context to 54.53 for Metis with that message stored, a 22.18-point gap. + +The latent representation also changes what can be inspected. A text record can be cited, corrected, permissioned, and deleted as a distinct object. A blended matrix does not naturally preserve those object boundaries. The paper identifies semantic confusion, capacity, controllability, and interpretability as open problems and explicitly describes hybrid native-external systems as future work. + +## What changes for agent architecture + +Metis shows that one layer of memory can move from an agent harness into the model without online gradient updates or repeated context replay. That is a real architectural option, especially for dense short-term personalization and long histories whose full text would be expensive to prefill repeatedly. + +It does not eliminate the rest of the memory lifecycle. Long-lived agents still need durable identity, source provenance, permissions, correction history, portability, restoration, and evidence about which state caused an action. Those are system properties represented by [context repositories](/knowledge/context-repositories), explicit records, and trajectories rather than by recall accuracy alone. + +The useful distinction is therefore not native memory versus external memory as mutually exclusive camps. It is latent learned state for efficient behavioral continuity alongside inspectable external state for custody and truth. Metis is evidence that the first half can work. Its own failure analysis is evidence that the second half remains necessary. diff --git a/knowledge/receipts/technical-publication/metis-memory-foundation-model.json b/knowledge/receipts/technical-publication/metis-memory-foundation-model.json new file mode 100644 index 0000000..5b27e7b --- /dev/null +++ b/knowledge/receipts/technical-publication/metis-memory-foundation-model.json @@ -0,0 +1,19 @@ +{ + "schema": 1, + "kind": "technical-publication-authorization", + "entrySlug": "metis-memory-foundation-model", + "route": "metis-memory-foundation-model", + "authorizedBy": "Cameron", + "recordedAt": "2026-07-31T19:02:42.509Z", + "scope": "technical-publication", + "authorizationBasis": "Telegram message 17325 linked the public Metis paper and explicitly requested a Public Knowledge page.", + "exactRenderReviewed": false, + "implementationReviewedBy": "Co", + "constraints": [ + "Use only the public paper, repository, model collection, and existing public Knowledge entries.", + "Attribute benchmark, architecture, and release claims to the authors or project sources and preserve the preprint evidence limits.", + "Export no private conversation, people, relationship, health, work, financial, housing, location, or operational context.", + "Do not represent technical-publication authorization as exact Cameron review.", + "Run deterministic, responsive, deployment, live-route, and protocol-mirror checks before reporting completion." + ] +} -- 2.51.2