Trying to remember everything made the agent worse

The test stored more conversations and working notes, pulled back anything that looked semantically related and added it to the next prompt. Similarity was reason enough to include something. What happened was that the retrieved material took up the space the agent needed to understand the current job. Old plans came back after the plan had changed, a phrase shared by two unrelated projects pulled in the wrong context, and tentative ideas from long transcripts came back with the same weight as settled decisions. The model had to work out what was current and relevant before it could even start, and a bigger context window would only hold more of the irrelevant stuff.

So I'm treating memory as a few types of record instead of one growing transcript. The boundaries might move as this develops:

• Identity: a small, deliberately edited set of things that hold across most sessions, like who the agent's working with, broad preferences and standing constraints. A conversational guess shouldn't end up here.
• Durable decisions: what was decided, when, in what scope and where the source is. Superseded ones stay traceable but stop being presented as current.
• Project state: what's true now, like the current branch, open problem, milestone or next action. It changes often and needs an owner or source.
• Searchable archives: transcripts, logs and documents that might matter later but don't get included automatically.

The retrieval sequence I'm testing is to work out the task and its project, load only the identity and constraints that task needs, fetch current project state, pull a few related decisions (preferring current ones) and search the archive only when the task or an unresolved reference calls for it. Semantic search still runs, just inside filters on project, record type and status. A close vector match from an unrelated project is still from an unrelated project.

Durable records keep a link back to their source (a file and section, an issue, a database record or a dated message), so the memory itself can stay short. Corrections are explicit too. One record supersedes another instead of being edited in place, which means I can still work out why an earlier session behaved differently, and a provisional interpretation gets labelled and either expires, asks for confirmation or stays out of durable memory.

I'm also setting a retrieval budget by count and by space, tight enough that the system has to rank. Each result should say why it was picked: record type, project, status, source, score and token estimate. If half the budget is coming from one old transcript you can see it in the trace. Without that trace, a weak answer just looks like the model's fault when the prompt was already full of irrelevant history.

The agent got worse because I tried to make it remember everything, and the fix is loading less, chosen by purpose, with each result saying why it was picked. Recall on its own can score well while the work gets worse, so I'm building small fixtures where the needed context is known, each listing records that should appear, records that must stay out and facts that need a source lookup. For anything that still loads by default, I want a plain answer to which tasks need it and how it'll be corrected when it changes. The next cleanup is shrinking the default identity context and moving changing facts into project-state records.