What Can AI Remember—and What Must a Work System Preserve?

Geoffrey Chen

After AI Enters the Workflow · Season One: “From Answering Questions to Participating in Work” · Article 4

An employee works with an AI assistant over several weeks. The assistant appears to know the project’s preferred terminology, the format of its reports and the manager’s usual concerns. One morning it prepares an update using an old budget, treats a proposal as an approved decision and omits a restriction agreed in a meeting two days earlier.

The employee is surprised: “I thought it remembered the project.”

That sentence captures a common mistake. We use the word memory for several technical mechanisms and then attribute to them the kind of continuous understanding that a colleague might possess. A model can receive a long conversation, retrieve stored notes or be given a generated summary. None of those mechanisms by itself creates an authoritative project record.

As AI becomes part of ongoing work, the question is not merely what the system can recall. The more important question is what the work system must preserve independently so that decisions do not depend on whether the assistant “seems to remember”.

“Memory” conceals several different mechanisms

At least four different things may be described as AI memory.

The first is the model’s trained parameters. Training leaves statistical regularities that allow the model to generate language and answer many general questions. This is not a database of current organisational facts, and it cannot be relied upon to contain a particular project decision or its latest version.

The second is the current context window: the instructions, conversation, documents and tool results supplied during an interaction. The model can use this material while it remains available, but the context is finite and may be truncated, summarised or replaced.

The third is retrieved external information. A system may search files, databases or stored notes and insert selected passages into the current context. This can create continuity across sessions, but its quality depends on what was stored, how it was indexed, what the retrieval query selected and whether the source remains current.

The fourth is a product-level personalisation or memory feature. A service may retain preferences or user-provided facts and use them in later conversations. Such features can improve convenience, yet they are designed for assistance and personalisation, not necessarily for records management, evidentiary integrity or legal retention.

These mechanisms can all make an interaction feel continuous. But continuity of experience is not the same as continuity of an official record.

A long context is not a reliable working memory

It is tempting to assume that if a model can accept a very large amount of text, the whole project can simply be placed in its context. Two problems follow.

First, capacity is not the same as effective use. Research on long-context models found that performance can vary according to where relevant material appears, with important information in the middle sometimes used less effectively. TACL, “Lost in the Middle: How Language Models Use Long Contexts”

Second, a large context does not express authority. It may contain an approved budget and an obsolete spreadsheet, a binding decision and a brainstorming note, a current rule and a copied extract that has since been superseded. If their status is not represented, the model has no dependable basis for deciding which one governs.

Long-running agents create an additional difficulty. Every observation, tool result and intermediate message competes for limited attention. Systems therefore compress history, write summaries or move information into external stores. Anthropic’s guidance on context engineering describes the need to select, compress and retrieve information deliberately rather than let an agent’s history grow without control. Anthropic, “Effective context engineering for AI agents”

Compression is useful, but it is also an editorial act. A summary can omit a qualification, merge a proposal with a decision or lose the reason an exception was granted. The more a workflow relies on generated summaries, the more it must preserve links back to the underlying records.

Retrieval does not automatically create a source of truth

Retrieval-augmented generation allows a model to consult an external collection at the time of generation. The foundational RAG work combined parametric model knowledge with a retrievable document index for knowledge-intensive tasks. NeurIPS, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”

This can make information easier to update and can support citations. Yet a retrieval layer remains a selection mechanism. It may fail to find the governing document, rank an older copy above the current one, return only the paragraph containing a keyword or omit a relevant exception elsewhere in the file.

A source of truth requires more than search. It needs ownership, version control, status, scope and rules for resolving conflict. A retrieved passage becomes useful evidence only when the system can identify where it came from, whether it is complete, when it applied and whether a later record replaced it.

The distinction is important because fluent synthesis can hide a retrieval failure. A model may give a polished answer from three secondary documents while missing the primary decision that controls the case. The answer appears well informed; the work system has nevertheless failed to preserve authority.

Work must preserve traceable state

What should a durable work system retain? Not every word of every conversation. It should preserve the state needed to understand and reproduce consequential work.

That normally includes:

  • the current objective, scope and owner of the task;
  • authoritative facts and the records from which they came;
  • decisions, including who made them, when and under what authority;
  • open questions, rejected alternatives and unresolved conflicts;
  • versions of important artefacts and relationships among them;
  • actions taken by people and systems;
  • approvals, exceptions and reasons;
  • the point from which work can safely resume.

This is a different object from a model’s conversational memory. It is an auditable representation of the work’s state. It should remain intelligible if the model changes, the assistant is disabled or a different person takes over.

NIST’s AI Risk Management Framework Core emphasises documentation, defined roles, monitoring and mechanisms for managing risks across the lifecycle. These governance practices matter because the reliability of an AI-enabled workflow depends on more than model output at one moment. NIST AI Resource Center, “AI RMF Core”

Personalised AI memory creates a different risk

Personalisation can be valuable. An assistant that remembers writing preferences, accessibility needs or recurring tasks saves users from repeating themselves. But personal memory creates questions that project records do not answer.

What information was retained? Was it directly stated, inferred or generated as a summary? Can the user inspect and correct it? How long will it remain? Is it used only for the user’s benefit, or also for product improvement, evaluation or other purposes? Can sensitive information be kept out of memory even if it appeared in a conversation?

These are privacy and identity questions as much as convenience questions. NIST’s Privacy Framework provides a way to consider how data processing affects individuals and how organisations can govern, control, communicate and protect that processing. NIST, “Privacy Framework”

A work system should not silently convert every conversation into persistent memory. Retention should have a purpose, an owner and a deletion rule. Users need a way to see and correct personal records, while organisations need to prevent a preference store from becoming an unofficial repository for contracts, health information or confidential decisions.

Someone must make memory expire

Most discussions focus on remembering more. Mature systems also need to forget correctly.

A former employee’s access, an expired price, a rescinded policy and a superseded decision should not remain equally available to future generations. Deletion is not always the answer: some records must be retained for audit or legal obligations. But they should be marked as historical and separated from the material that controls current action.

Responsibility for expiry cannot be left to the model. A system needs record owners, review dates, supersession relationships and rules for archival, deletion and access. When a generated summary is updated, the previous decision trail may still need to remain. When a personal memory is removed, derived stores and indexes may require attention as well.

The question “Does the AI remember?” is therefore incomplete. We also need to know who decides what remains active, what becomes historical, what must be deleted and how those changes propagate.

Conclusion: continuity cannot rest on the impression that AI remembers

AI can make work feel continuous. It can carry instructions across a long context, retrieve earlier records and adapt to stored preferences. These are useful capabilities, but they do not relieve an organisation of maintaining its own evidence and state.

The governing principle should be:

Anything necessary to explain, continue, verify or reverse consequential work must be preserved in a system of record that does not depend on a model’s apparent recollection.

AI memory can support that system by retrieving, summarising and presenting relevant information. It should not replace the authoritative record. A new model, a lost conversation or a mistaken summary must not erase the basis of a decision.

The goal is not perfect recollection. It is dependable continuity: current facts can be distinguished from history, decisions can be traced to authority, users can correct personal information, and work can survive the assistant that helped perform it.

Primary sources and further reading

Continue reading: Explore the After AI Enters the Workflow series.

Discover more from Geoffrey Chen

Subscribe to get the latest posts sent to your email.