Five ways my nightly agent system failed while every run looked successful

One morning I opened a plan that had been written overnight. It was detailed, internally consistent, and technically credible. It specified the machine, networking, storage, ingress, monitoring components, and order of deployment for a Kubernetes observability stack.

The stack had already been running for weeks.

Nothing had crashed. The agent completed its task. The output looked like good engineering work.

That is the failure mode that has cost me the most time since I started running agents unattended: not agents that stop, but agents that reach the end from the wrong premise.

The distinction underneath every case below is simple to state and easy to skip in practice: a process finishing is not proof that the operation it stood for actually happened. I now check three things separately — liveness (is it running), capability (can it actually do the thing), and outcome (did the result land where it was supposed to) — and a task only counts as done when the outcome check passes. The first two are necessary, never sufficient.

What the system actually is

Agents are no longer a tool I open for an isolated task. During the day I use them interactively across software engineering, data workflows, infrastructure operations, research, and documentation. When I step away, recurring routines continue fixed pipelines or pick up work that was deliberately queued for them. The nightly part is just the unattended mode of the same agents.

That setup sits inside a real homelab rather than a demo environment. Two physical hypervisor hosts carry more than thirty services across dedicated Linux containers, virtual machines, and Docker; one of the two is dedicated to GPU work, serving local models with Ollama. A separate Kubernetes lab carries only what I’m building and breaking there; the services my household actually depends on stay on Docker.

The executor is an agent harness running inside its own Linux container on one of these hosts — Claude Code at the moment, though the seat is deliberately swappable for another harness. It is not a restricted chat interface: it has a shell, read-write access to a structured filesystem knowledge base, Git repositories, shared skills, sub-agents, web access, and selected MCP connectors — capabilities declared per routine rather than assumed globally. The knowledge base is my maintained second memory — documents, project state, decisions, and code — versioned with Git and inspectable with ordinary tools.

Work reaches the system as fixed missions — routines that run the same pipeline each time — or through an inspectable directory of queued action files. A phone gateway accepts text and voice requests — voice transcribed locally — and delegates anything touching the private corpus to restricted gateways. Those remote gateways have no shell; the executor does. Security claims become meaningless when they are copied from one boundary and applied to the whole system.

The system is worth writing about for one reason: it works without somebody watching, against data I care about, and a capable agent will often continue toward its goal when a preferred path breaks. That is useful, and it makes failure harder to see. Night is not the architecture; it is the reliability boundary. In an interactive session I can notice a missing tool or a strange premise and intervene. An unattended run has to carry enough evidence to challenge its own success.

Five failures that returned plausible success

1. The MCP process was connected; its runtime had disappeared

I originally declared an MCP server through uvx package==version — convenient interactively, where uvx builds a disposable environment for you, and a poor fit for a long-lived server process.

The process kept running after the disposable environment beneath it had been garbage-collected. Modules that were already loaded still worked, so tool discovery responded and the client displayed Connected. The first tool call that required a lazy import failed with an error such as:

No module named 'rich.traceback'

When I found the problem, every long-lived MCP process on the machine was in this state. The transport was alive. The capability was dead.

The durable fix was a persistent installation with a stable executable path, plus a preflight that checks whether a running process still has a real environment underneath it. A green connection indicator is not a capability check; a representative tool call is.

2. The collector failed; the reporting pipeline forced success

A third-party API split one contract into two: search moved to a new host and path and switched to camelCase; detail requests stayed put with snake_case; the retired search endpoint began returning 410 Gone. Two parts of one migration — though one did not cause the other, as the first version of this article incorrectly claimed.

The more important failure was one layer higher. The collector did raise an error, but the routine formatted its output through a shell pipeline shaped like this:

collector --all 2>>run.log | grep 'summary' || true

The || true kept the report moving. The collection section came out empty, the later stages ran against old state, and the overall brief was still produced. The API had failed loudly into a file nobody was reading; the user-facing workflow had converted that into silence.

The adapter now treats search and detail as separate contracts, tested against captured payloads from both. The operational rule is broader: preserve a producer’s exit status before filtering its output for presentation. Formatting should never get to decide whether the work happened.

3. The files were readable; 22 of 23 Git repositories were not

The executor sees a data tree mounted into its container, with files from different ownership contexts, while unattended routines run under a different Unix identity.

Ordinary file reads worked. Git did not. Its ownership protection rejected 22 of the 23 repositories as potentially unsafe with dubious ownership: a routine whose mandate included comparing written plans with repository state could still search and read files, but git status, git log, and git diff failed across almost the entire corpus. The agent continued and wrote a plausible report from what it could see.

This was not a reason to disable Git’s protection globally. The fix was a bounded safe.directory policy for the personal data mount, declared in the machine installer, followed by an inventory check that counts any repository Git still cannot open.

“The repository is mounted” was a storage fact. “The agent can inspect its state with Git” was a capability claim. I had tested the first and assumed the second.

4. The evaluator exited zero and wrote zero records

One routine exports a batch of records, asks a headless agent to evaluate them, and expects structured results back in SQLite. On one run, the agent could not execute the command it needed because the tool permission and the command form did not match.

Its harness nevertheless exited 0. The generated brief announced that the batch had been evaluated. The database contained zero new evaluation rows.

The routine now counts the rows actually persisted after the agent exits. If the input batch was non-empty and the written count is zero, the brief reports a hard failure regardless of what the harness returned or what the agent wrote.

The exit code belongs to the harness, not to the work: it reports that the agent session ended without the harness itself crashing. The model has no exit code of its own — when it gives up, it says so in prose, and prose is not a status.

5. The plan was good; its world did not exist

The plan from the opening proposed a first monitoring deployment for my personal Kubernetes lab: machine shape, load balancing, ingress, storage, Prometheus, Grafana, Alertmanager, backups, and a phased execution order.

The reasoning was often good. The premise was not: the core of that stack was already deployed and healthy. The run had planned from maintained documentation and assumptions — it even said so — and the documentation described an earlier state of the lab. Nothing in the workflow required observing the cluster first, so the plan reconciled a target against a stale model of the world, and the real remaining work — DNS, backups, scrape coverage, alert routing — never surfaced.

Declarative infrastructure solved this long ago: a controller never trusts its stored model, it reconciles desired state against observed state on every loop. My planning workflow had a target and documentation, but no observation step. The executor had the access — data, repositories, a shell, connectors — and access alone does not create evidence; the workflow has to make the agent use it.

I now treat an infrastructure plan as invalid unless it opens by separating what was observed directly, what came from documentation, and what remains an assumption.

The shape they share

Each failure had a signal that was true at one layer and useless at the layer I cared about.

Local signal	What it actually proved	Missing outcome check
MCP displayed Connected	A process and transport responded	A representative tool call succeeds
The daily brief was generated	The reporting shell reached the end	Fresh records were collected
The data volume was readable	File bytes were accessible	Git could inspect every expected repository
The evaluator returned 0	The harness process ended normally	Evaluation rows were committed
The plan was coherent	The reasoning was internally consistent	Its premise matched the running system

No individual component owned the distance between those signals and the outcome I actually wanted.

Agents make that distance more subtle than classic automation does, because persistence is part of their value. A blocked tool leads to another path; a failed command still gets summarised; incomplete evidence still becomes a coherent answer. Hard failures turn into plausible degradation — the run still produces output, and the output still looks authored. A human can mistake fluency for evidence.

None of this called for a new engineering practice. The incidents cross package lifecycle, API contracts, data pipelines, Unix ownership, transactional state, and infrastructure discovery — and everything that fixed them is ordinary operations discipline: monitoring that asserts on outcomes rather than on process health, tests that run in the identity production runs under, pipelines that fail loudly. Agents needed the old discipline applied at a new boundary, where fluent output is too easily accepted as evidence.

The rules I use now

Separate liveness, capability, and outcome. A process answering, an operation succeeding, and the state change appearing at the destination are three different facts. Important workflows check all three.

Preserve failure before presentation. Capture the producer’s status and validate its output before piping, filtering, summarising, or asking another agent to continue.

Assert on domain state. Count the database rows. Open the generated file. Check the destination inventory. A successful exit is not a substitute for the state transition the process was hired to make.

Test tools in the identity that runs unattended. Interactive access from my laptop proves nothing about ownership, PATH, credentials, or tool allowlists inside the server-side executor.

Observe before planning. A plan separates direct observation, documentation, and assumption — and starts from observed state, not from the last written description of the system.

Describe security boundaries precisely. My executor has a shell. The phone-facing gateways do not. Each routine receives an explicit tool surface. Collapsing those facts into “the agents have no shell” would be simpler and false.

Added after publication. A reader sharpened the third rule, and they were right: an outcome assertion should be a bounded count, not a presence check. A table exists whether or not anything was written to it, and a count above zero still passes when the same batch is processed twice. Asserting that a run wrote between N and M rows catches the silent zero and the double-run in one check. That is the version I am moving my routines to, and it is the one worth copying.

I still use the system, and the agents are still worth it. They turn maintained context, tools, and spare machine time into useful work. I just read “completed successfully” much more narrowly now: it means a process reached the end. The outcome still has to prove itself.