Operating local AI infrastructure after the migration

Right after a migration, the system can look healthy while I've still got shells open, environment variables loaded and a couple of services I started by hand. The next reboot wipes all of that, which makes it the quickest way to find out whether the machine can come back without me remembering a private sequence of commands. So I want every component managed by the OS or the container runtime, with a restart policy and a known service user. The agent runner shouldn't accept work until its database, memory store and local APIs are ready. If the memory service doesn't come up within a time limit, the agent should stay unavailable or go into an obviously reduced mode. Quietly starting without memory means it's doing work under different conditions while the dashboard says everything's fine.

Paths are the other thing that bites after a move. Prompts point at supporting files, tools expect working directories and backup scripts target database volumes, so a changed mount point can break all of them without touching any code. I'm keeping authoritative data, config, generated indexes, caches and logs in separate places, with the mappings in versioned config rather than an ad hoc launch command, and I'm testing permissions as the service account (not my own user). A handy check is to create a throwaway record through the running service, confirm where it actually landed, then delete it through the same interface. That catches a service happily writing to an empty volume while the existing data sits somewhere else.

Backups are where I trust exit codes the least. A copy taken while the database is writing can be inconsistent, an encrypted archive is useless without its key and restore command, and retention can keep plenty of copies of the same corruption. I want each job to log the source, start and finish times, size, checksum, encryption method and retention action, and then I want a monthly restore into an isolated environment. The restore should cover what can't be recreated (durable memory, source links, agent config and activity history) and answer a few fixed queries, including one that follows a memory record back to its source. Derived indexes can be rebuilt, though the rebuild needs testing too.

For health checks, a single 200 OK tells me the web process answers HTTP and nothing about the queue, database, disk or model endpoint. I'd rather have small checks at each layer plus one harmless synthetic task that crosses the main path, something like a read-only job that retrieves a known test record and writes an activity event to a test namespace. Each check gets a timeout and an owner, and I'm carrying a task ID and run ID through every component so I can see which retry called which tool without grepping timestamps. Logs keep IDs, durations and error classes (no prompts or credentials), with size limits so debug output can't fill the disk.

My weekly check is short: failed and retried jobs, backup recency, disk growth and expiring credentials, plus any unexpected restarts.

Operating this setup after the migration mostly means making sure the machine doesn't depend on me: services start in order on their own, data lands where I think it does, backups actually restore and failures show up where I'll see them. The next concrete thing I'm doing is a cold reboot followed by that synthetic read-only job. I want the run record to show every service became ready in order, the known memory record came back and the result was written without me stepping in. Anything that needs a hand from me during that run is the next thing I'll fix.