What I am testing in long-term AI memory

Saving every conversation gets you a big searchable archive, but the archive can't tell the agent which statement is current, whether something was a decision or just a suggestion, who it applied to or where it came from. Most chat messages are coordination or wording that was useful for a few minutes. So I'm starting with a narrow memory model and the questions a later session actually needs answered.

The records I care about have a clear future use. A decision stops the same question being reopened, a constraint stops a proposal the project has already ruled out, and a source reference leads back to the evidence instead of a loose paraphrase. Each record declares its type and scope. "Use Australian English" is a durable writing preference. "Try the second query" is probably session state. "The integration uses version 2 of this endpoint" might be project knowledge, but it needs an effective date and a source because the endpoint can change. I'll get some of these classifications wrong, so I'm keeping the original source link and letting records be corrected or superseded.

The graph is there so the relationships are explicit. A decision connects to the project it affects, the person who approved it, the source that supports it and any later decision that replaces it. If a new session asks why a project uses a particular data boundary, retrieval can start at the project, follow the active decisions and return the source, whereas a text search for "data boundary" would also drag in abandoned options and unrelated projects. I'm keeping the relationship vocabulary small (APPLIES_TO, DECIDED_BY, SUPPORTED_BY, SUPERSEDES), each with a defined direction, and using stable internal IDs so two entities don't get merged just because their names look alike. Once the graph gives me a bounded neighbourhood, semantic or keyword search can rank what's inside it.

Time gets its own fields. created_at is when the record entered the system, effective_from is when the decision started applying, and effective_to or a superseding edge says when it stopped. Inferred records are labelled as inference so they don't quietly turn into approved decisions after being retrieved a few times, and access follows the source, so a restricted document can't come back as a broadly visible summary. For the evaluation I've got a small set of questions phrased the way a future session would ask them:

• What decision currently governs this project setting?
• Who approved it, and where is the supporting source?
• Has the earlier decision been replaced?
• Which stated preference applies to this output?
• Is there no durable memory for this question?

That last one matters because the right answer is no memory at all, and returning the closest bit of old text would just add weak context. For each question I'm recording whether the right memory came back, its rank, whether stale or unrelated records showed up and how much context was returned. Then I'll run the same questions again after adding distractor records from other projects and older decisions, to see whether scoping and supersession hold up once the graph gets messier.

The first test after that is small: take one decision with a source, retrieve it, supersede it and ask the same question again, and check that the new decision comes back with a clear path to why it changed. That's what I'm testing in long-term AI memory, whether typed and scoped records connected to a person, a project and a source can give the next session the current answer and its evidence, which a saved archive of conversations can't do on its own. The results from those question runs, with and without the distractor records, will tell me which record types are worth keeping for the next session and which ones are just making retrieval noisier.