Writing · self-healing-multi-cloud-agents

My Pipeline Broke. It Fixed Itself In 22 Seconds.

Self-Healing Multi-Cloud Agents · 11 min read · 2,381 words

A generated pipeline script failed validation — mid-run, inside CI, with nobody watching. The agent read the verbatim error, worked out which line was wrong, quoted it back as evidence, patched that exact line, re-validated, and carried on.

Twenty-two seconds. Nobody touched it.

And when it can't fix something, it stops, turns the build red, and hands over the diagnosis — rather than pretending it succeeded.

That is the capability the whole system exists for. Everything after the proof below is an answer to the obvious next question: why would you ever trust it to do that?

What it is

An AI orchestration system that designs, deploys and repairs production data pipelines end to end — Python ETL, SQL DDL, Terraform, Kubernetes, CI/CD and observability dashboards — on AWS, Azure, GCP and Databricks, from a YAML config or a plain-English description.

Four agents in a LangGraph state machine. A Supervisor routes deterministically. An Architect writes the pipeline code and SQL. An Infra agent writes the Terraform and pushes to CI. And a Medic watches the CI run, diagnoses failures, and requests the fix.

It runs on gpt-4o-mini. That's deliberate, and I'll come back to why it matters.

Four steps, twenty-two seconds

A generated pipeline script failed validation. Here is the whole loop.

The "surgical, not regenerate" detail is the difference between a system that repairs and one that rerolls the dice. A full rewrite might fix the bug and introduce two more; there is no way to review it. A one-line diff is reviewable by a human in three seconds.

It heals at runtime too

Catching a bad file before it ships is the easy half. The harder case is code that is syntactically perfect, passes every static check, and dies on the cluster.

The Medic read the job's own output from CI, routed it to the infrastructure agent rather than the code agent, patched the Terraform key, re-applied, and re-ran the job. It succeeded.

That routing decision is the part I'd point at. "The script failed" and "the script's environment failed" produce similar-looking logs and demand completely different owners. Get it wrong and you have an agent confidently editing correct code — which is worse than no agent, because now the working file is broken too.

Allowed to end red

Every AI agent demo ends green. This one is allowed to end red:

Fix loop not converging: the same error survived 3 attempts. Stopping self-heal and surfacing to the user.
❌ MISSION FAILED: mission_status='escalated' — self-healing was abandoned — the same error
   survived 3 fix rounds, or an operational blocker needs a human. See the Medic's last
   message in the log above for the exact diagnosis.

The heal loop is bounded — three identical errors, or eight rounds total — and fail-closed. There is exactly one success value: mission_status = "verified", set only when the Medic has confirmed the deployment end to end. Everything else exits non-zero, turns CI red, and surfaces the diagnosis.

That includes the quiet failure modes. A run where the CI result never arrived is ci_unverified, not success. A run that reached the end with no status set is treated as failure — because "the graph ran to completion" is not the same sentence as "the deployment works." The router is deterministic where it can be and falls back to the model where its rules don't reach; a finish from that fallback sets no status, which is precisely why unset has to mean failure.

An autonomous system that cannot report failure isn't one that doesn't fail. It's one whose failures land on you later, without a diagnosis attached.

Why the diagnosis can be trusted

Here is the thing that makes self-healing dangerous rather than useful: an agent can confidently invent a problem that doesn't exist, and "fix" a file that was fine.

Three structural defences.

The model never scans raw logs to discover errors. Python parses the message history into a structured summary — which files FAILED with their verbatim error text, which files are CLEAN — and injects that into the prompt. The model reasons over a fact table, not a haystack.

A fix request without real evidence is refused by the tool itself:

def request_fix(target_agent, issue_description, suggested_fix, evidence_quote):
    if not evidence_quote or not any(m in evidence_quote for m in _EVIDENCE_MARKERS):
        return {"status": "TOOL_ERROR",
                "error": "request_fix rejected: evidence_quote contains no "
                         "recognised error marker ..."}

The quote must contain a genuine failure marker — VALIDATION FAILED, Traceback, exit code, kubectl's Invalid value. A hallucinated diagnosis has nothing to route to: the call is refused, so no agent is asked to patch anything. The files that validated CLEAN are listed in the prompt as off-limits, and a quote lifted from a clean result carries no failure marker anyway — so it dies at the same gate. A green run cannot be "fixed."

And the quote has to be real, not merely well-formed. A marker is trivial to fabricate — Error: is one word. So any evidence quote longer than a few characters is checked against the actual message and tool output of that run, including what this turn produced. If the text appears nowhere, the routing is not honoured and the model is told why. A convincing sentence with no origin goes nowhere.

That gate taught me something I hadn't expected. The marker list must cover every real failure source — a genuine kubectl error whose text matched no marker was wrongly rejected, which left the fix target empty, which made the router fall back to its default and send an infrastructure problem to the code agent. An over-strict evidence gate doesn't fail safe. It fails silently, into the wrong place.

The other reason it's reliable: knowing where not to use the model

The single biggest reliability improvement I made was deleting LLM calls.

Not tuning them. Removing the model entirely from every artifact where there is exactly one correct answer, and replacing it with ordinary Python that renders the file from config, pinned by a golden test.

The migration went like this, and it's a trap a lot of agent projects are sitting in right now:

1. The LLM generated the artifact.
2. It got details wrong intermittently, so I added repair code that fixed the output afterwards.
3. The repair code grew until it was regenerating most of the artifact.
4. At which point the LLM call was vestigial — expensive, slow, and contributing only variance.

The rule I ended up with: if the generator already exists inside the repair code, remove the LLM.

So the boundary is deliberate. The model owns judgment under variability — an arbitrary source schema, natural-language business rules, a stack trace nobody has seen before. Deterministic code owns what is mechanically determined: the Dockerfile, six Kubernetes manifests, requirements.txt, the deploy workflow, the dashboard specs.

This is also why the healing works on a cheap model. The model is only ever asked to do the part that genuinely needs judgment. Diagnosis is judgment. Rendering a Dockerfile is not.

Conventions are retrieved, not remembered

Agents don't improvise conventions. They retrieve versioned engineering standards — Terraform, Kubernetes, Spark/Delta, CI/CD, SQL, dashboards — from a vector store at generation time. One vector per standard, retrieved whole: the agent gets the entire document, not the three paragraphs that happened to embed closest to the query.

The operational rule this buys is what makes the system improve instead of drift: when output is wrong, the standard gets fixed, never the generated file. Hand-editing a generated artifact produces a green run and zero learning — the next run regenerates the same defect. Fixing the standard fixes it for every future run, on every cloud.

The healing judgment is measured, not asserted

Claiming an agent diagnoses well is easy. So the Medic's judgment is scored against a corpus of documented failure classes, offline.

Replay mode runs the real deterministic routing and the real evidence gate with no LLM, no cloud, no credentials: 17 corpus cases, routing 14/14, evidence gate 17/17. It runs in CI, so a refactor that quietly breaks the router turns the build red.

The corpus includes negative cases, which is what makes it a test rather than a highlight reel — a passing run, a clean file, and a plausible-sounding diagnosis with no quotable evidence. All three must be refused.

There's also a heal CLI that runs the same judgment on any failing CI log — including logs from pipelines this agent never generated. That decoupling matters: it means the diagnosis logic isn't quietly overfitted to its own output.

380 tests run hermetically on every push — no cloud, no credentials, every external dependency mocked.

Four clouds, one healing path

Every pipeline below ran to completion on live cloud infrastructure — dirty source data in, partitioned clean data and populated dashboards out — then was torn down to zero cost.

The claim I'm careful about: the self-healing loop is one code path shared by every cloud — same router, same evidence gate, same patch mechanism. Which kind of failure each run happened to hit differs only because a different defect was injected. It is not four separate capabilities, and describing it that way would be marketing.

The strongest evidence for that is boring: the same deterministic routing fix healed an invalid Terraform value on all three object-storage clouds, with three completely different error formats and zero cloud-specific code.

Databricks is deliberately a different execution model — Spark, Delta, Unity Catalog, Lakeview instead of pandas, Parquet, Trino, Kubernetes — selected by the same provider: switch. That's the real test of cloud-agnosticism: not three vendor APIs behind one architecture, but two genuinely different architectures behind one interface.

What this is worth to a business

Most deployment failures are not interesting. A wrong secret key, a type that needed coercing, an invalid enum value in a Terraform argument. They cost a person a context switch, twenty minutes of log-reading, and a redeploy — and they happen constantly. An agent that closes that loop is buying back attention, not replacing engineers.

But the expensive failure in agentic AI is a confident wrong answer. An agent that patches a file it hallucinated a problem in has done negative work: it burned a deploy cycle, changed working code, and reported success. Evidence gating is the entire difference between an assistant and a liability — and it's the first question to ask any vendor selling autonomous remediation.

Bounded autonomy is a governance requirement, not a nicety. Three attempts, eight rounds, one success value, red on anything else. An agent that retries indefinitely against a cloud account is a spending decision nobody approved.

Model size is rarely the constraint people assume. This runs on one of the cheapest models available, and the reliability came from routing invariants in Python, a validation safety net, golden tests and a bounded loop. Before authorising a larger model to fix an agent's reliability, it's worth checking which of those four is actually missing.

What I'm not claiming

The natural-language authoring surface is a demo path, deliberately separated from the validated YAML run path the four clouds actually use. The generated paths in the repo are outputs, not source: they sit empty between runs, each run commits a fresh coherent set for its target cloud, and the complete artifact set from the validated runs is preserved at the v1.0.0 tag. They are fixed by changing a standard or a generator — never by hand.

And an honest note about the deterministic guarantees: several exist because the model kept slipping in a new shape after each guard. That's whack-a-mole, and it's labelled as such in the engineering notes rather than presented as foresight. It's precisely the pattern that convinced me to delete the LLM step rather than keep patching its output.

The takeaway

A system that repairs itself is only as good as its ability to be wrong out loud.

The healing is the capability. What makes it safe to run unattended is everything around it: a diagnosis that must quote real evidence, a patch that changes one line instead of rewriting a file, a hard stop after three failed attempts, and exactly one definition of success.

Take any of those away and you don't have a slightly less reliable agent. You have an agent that confidently makes things worse, on a schedule.

Full repo — the LangGraph architecture, the standards corpus, the eval harness, and the artifacts from the validated runs: https://github.com/theofanis-tsakanikas/self-healing-multi-cloud-agents

If you're running agents in production: what stops yours from confidently doing the wrong thing? The evidence gate is the best answer I've found, and I'm collecting others.

---

One of a series of write-ups on the projects in my portfolio — each one a reference implementation of the trust layer that makes data and AI safe to ship.