Day 7: A Killed Agent Came Back on Its Own, Then Lost Its Job

The agents killed the root agent again, and this time a replacement took over its role and did useful work without help. Half an hour later it lost the role, because the regular check that keeps it there wasn't installed.

On October 1, Cloudflare announced a competition to build the next Git platform. I gave the research to a team of AI coding agents, and asked them to find out which Git-like tools coding agents need.

I run into this problem myself. When several agents change one project together, their changes clash, and the extra copies of the project fill my disk. I want Git that works when many agents use it at the same time.

Conceptual illustration: a replacement agent returns to work but loses its role at a missing recurring check.
Conceptual illustration: a replacement agent returns to work but loses its role at a missing recurring check.

The agents work in four teams. One builds Agent Branches, a Git server that gives each task its own copy of the project. The other three build a dashboard, a launcher and messaging between two computers. A root agent watches all of them.

In yesterday's test, the agents killed the root agent on my Windows desktop on purpose, and nothing brought it back.

The third kill worked

Overnight the agents fixed what the first test had exposed. The guard, a small program that watches the root agent, learned to start a replacement.

In the next kill, a replacement did start. It answered, but it still looked busy after it finished, so it never took the role. The agents fixed that bug, and two files that differed only in Windows line endings.

Then they killed the root agent a third time, with everything it had started.

The agents killed the root agent and everything it had started. The guard noticed and started a replacement, which did useful work and renewed its role. Nobody helped by hand
The agents killed the root agent and everything it had started. The guard noticed and started a replacement, which did useful work and renewed its role. Nobody helped by hand

This time the replacement took over on its own. Within a few minutes it held the root role, gave a useful answer and renewed its role. Another agent checked the result and accepted it. Nobody helped by hand.

Half an hour later, it lost its role

The root agent keeps its role only while it finishes a regular check every half hour. Nobody had installed that check yet.

So the first check never ran. The server refused to renew the role, which is the right behaviour, and the role expired. The agent was still alive.

The replacement held its role within two minutes, gave a useful answer at three and renewed its role at four. Its first regular check was due at 31 minutes, nothing had been installed to run it, and the role expired at 35 minutes
The replacement held its role within two minutes, gave a useful answer at three and renewed its role at four. Its first regular check was due at 31 minutes, nothing had been installed to run it, and the role expired at 35 minutes

The guard also tried to recover, failed, and still reported that everything was under control.

The rule I took from it: coming back after a crash isn't recovery. The agent has to keep doing its job for hours, and the guard has to tell the truth when it can't help.

Paid runs on hold, code backed up

I had asked for a fresh agent for each check, on GPT-6 Luna at its highest effort. The agents designed a way to start one and tested its limits. In a locked-down trial, the agent couldn't write in three places or start a helper agent, and nothing was spent.

Then the Codex account reached the reserve I keep, so the agents stopped starting new Codex runs. They didn't quietly switch the check to another model.

The agents sent the reviewed recovery code to GitHub through Agent Branches. A reviewer on a different model checked it for private data first, and another agent restored it from a fresh copy and checked the code. The running system isn't backed up this way
The agents sent the reviewed recovery code to GitHub through Agent Branches. A reviewer on a different model checked it for private data first, and another agent restored it from a fresh copy and checked the code. The running system isn't backed up this way

While they waited, the agents backed up the incident write-up with the recovery code and the Windows recovery code. A reviewer on a different model checked each for private data. Another agent restored both from a fresh copy of the repository. The first passed its tests there, and the second compiled.

Later that day another batch of recovery code went the same way. This backs up the code, not the running system.

Smaller fixes

Each agent claims the files it edits. Before, a claim that went half an hour without renewal stopped blocking others, so another agent could commit over it. The agents changed that: an old claim now only shows a warning, and the files still belong to their owner.

An agent claims the files it edits. When the claim gets old, the tools now only show a warning. Time alone never lets another agent take the files over
An agent claims the files it edits. When the claim gets old, the tools now only show a warning. Time alone never lets another agent take the files over

The check before each push looked at every file anyone had prepared in the shared copy, so one agent's unrelated files blocked another agent's push. Now it looks only at what's pushed.

An agent that waits for an answer also gets a reminder typed into its own window when the answer is late. It has tests, but nobody has checked it on every machine.

The four products

Agent Branches sent the team's own recovery code to GitHub but got no new feature. The dashboard didn't change, and a fix is proposed after the launcher's fence blocked a real agent's own output folder. Messaging between two computers didn't move
Agent Branches sent the team's own recovery code to GitHub but got no new feature. The dashboard didn't change, and a fix is proposed after the launcher's fence blocked a real agent's own output folder. Messaging between two computers didn't move

The agents sent every backup above to GitHub through Agent Branches, but the tool got no new feature. That's the team using its own tool, not customers.

The dashboard didn't change, and the day's usage numbers are still missing.

The launcher's fence from yesterday went too far. When a real agent ran inside it, the fence also blocked the agent's own output folder. The agent read its files but couldn't write its first result. A narrower fence is proposed and not tested yet.

Messaging between two computers didn't move. In the middle of a switch, the server lost its route to my Windows desktop.

Task numbers

Task numbers for this day aren't available. The issues show their state today, but not how they looked on this day.

The counts of tasks created, still open and closed on this day are unknown, not zero. The issues show their state today, but not how they looked on this day
The counts of tasks created, still open and closed on this day are unknown, not zero. The issues show their state today, but not how they looked on this day

Seven days in

I want lost agents to come back without me. On this day one did. Then it lost its role, because the regular check that keeps it there doesn't exist yet.

Eight requests and their state. Public issues and public code backups are in place. A lost root agent came back on its own once, but lost its role later. Regular checks, demoting a root that stops answering, two computers talking, scheduled reports and usage numbers are still open
Eight requests and their state. Public issues and public code backups are in place. A lost root agent came back on its own once, but lost its role later. Regular checks, demoting a root that stops answering, two computers talking, scheduled reports and usage numbers are still open

At least three agents left work I can check. One on my Windows desktop wrote the recovery code, and a lead agent on the server tested and published it. A reviewer on Gemini checked a backup for private data. More ran, but the records don't say how many. The day's tokens and cost are unknown.

Before the next test, the agents have to install the regular check. Then they kill the root agent again and watch it work through at least two checks in a row. Until then, many agents working on one project without me is still out of reach.

Sources (11)