I let my GPU workers shut themselves down after five minutes. Their decisions still had to be auditable.

The worst failure in my agent cluster was not that a model hallucinated. It was that the verifier turned the hallucination into a fact.

The setup was simple. I wanted to test a hybrid GPU cluster. The stable part lived on my local machine, with 4 NVIDIA Ampere GPUs with 24 GB each. The disposable part lived on AWS: a GPU worker that started on demand, joined the cluster over WireGuard, and shut itself down after five minutes without work.

To test it, I needed something more real than "hello world". I chose a whodunit-style forensic investigation: a team of six AI models investigating a real incident in the cluster itself. An orphaned process was holding a port, and neither Kubernetes nor the runtime could kill it. I gave the team the raw systemd journal logs - without the conclusion - and roles: a forensic analyst, a skeptic, an expert witness, a junior witness whose only job was to separate facts from guesses, and a judge.

As an infrastructure test, the meeting worked. All six steps of the ceremony - that is what the Underpass Choreographer calls an agent meeting with steps and rules - completed without errors. I did not see what happened live. I found it later, reading the records and traces that Choreographer leaves behind after each deliberation.

The forensic analyst, a 30B model, claimed that, "according to the spec", the pod had privileged: true. The material I gave the team did not say that anywhere. It made it up.

The junior witness - the smallest model in the team, 3B, whose only job was to check facts - recorded the invented claim as fact.

And the judge returned a high-confidence verdict, citing that evidence.

Then I ran an appeal: two model councils, three rounds of adversarial cross-review, with the correct answer to the incident already revealed. It got worse. Each round of "criticize your peer's proposal" behaved like "elaborate more". By the third round, the councils were proposing "semantic normalization layers with PCA/UMAP" and "sub-20Hz noise thresholds" - proposals that no longer had anything to do with the incident. Nobody went back to the evidence.

The problem was not how many voices my agent meeting had. The problem was that no claim was forced to rest on evidence - and the whole social machinery of the team, critique, review, verification, judge, amplified fluency instead of truth.

So I changed the question.

If agents are going to take part in real decisions, keeping the final answer is not enough. You need to be able to reconstruct how the answer was reached.

1. The Wrong First Question

My first question had been: "how many agents do I need for this to deliberate well?" It was the wrong question. The records show it in three places:

• The 6-model team - including two models served from an AWS g6.xlarge node securely connected to my local cluster - got the class of the root cause right, but fabricated the key evidence along the way. The verification mechanism accepted the fabrication instead of filtering it out.
• Three rounds of adversarial peer review made the conclusions worse with every iteration: 12 critiques and revisions, 31 LLM calls, trace a5ed9c25....
• Consensus is not evidence. A vote between models is still only a vote.

Recent literature describes exactly the failure pattern I was seeing in my agent deliberations:

After reading those, the fix could not be more rounds or better prompts. It had to be structural.

The operational conclusion was not "I need better models". It was:

The problem is not that agents hallucinate. The problem is letting unsupported claims become decisions.

2. The Right Question

Can a cluster with ephemeral workers run an agent decision and preserve enough evidence - traces, logs, metrics - to reconstruct the decision after those workers have shut down?

Two principles show up throughout the design:

Compute is disposable. Evidence is persistent.

The judge is not another model. The judge is a set of rules.

3. The Architecture

The architecture has three parts. A stable part coordinates. A disposable part adds GPU when needed. And the evidence lives somewhere else, in systems that do not disappear when the worker shuts down.

Local control plane

Stable

• 4x NVIDIA Ampere GPUs · 24 GB each
• Underpass Choreographer
• llm-gateway routing
• Prometheus · Tempo · Loki

WireGuard tunnel

Ephemeral GPU worker

AWS g6, L4

• Starts on demand
• Joins the cluster
• Serves model traffic
• Stops after 5 idle minutes

Telemetry outlives compute

Persistent evidence layer

• Traces: Tempo
• Metrics: Prometheus
• Logs: Loki
• Decision JSON in S3

The important parts:

• The worker shuts down on real idleness, not on a timer: a reaper - an automatic shutdown process - checks the Prometheus metric vllm:num_requests_running for the vLLM instances hosted in the cloud. If five minutes pass with no requests, it stops the instance.
• Choreographer captures the observability data: the evidence layer exists because Underpass Choreographer instruments every ceremony: one trace per decision, one span per step, structured events with ceremony_id, and metrics. Prometheus, Tempo and Loki keep that evidence outside the worker.
• The worker's public IP changes on every start; the Kubernetes control plane resyncs it. The model weights live in S3 Glacier Instant Retrieval, so the worker can be started from scratch.

4. The Experiment Rule: Nothing Enters Without Evidence

In the artifacts, this configuration is called evidence_bound_review. In plain English: the models could propose and argue, but the final decision could only include claims that pointed to a concrete item in the evidence pack. The names in the diagram are the YAML names; the flow is what matters:

evidence_collector
prepares the evidence pack

then

primary_engineer
proposal + evidence_refs

review

skeptical_reviewer (L4)
objections / review

revise

deterministic_judge
does each evidence_ref exist in the pack?

yes → enters the decision record

no → reject / ask for more evidence

The rule-based judge is not a model. It applies the same output contract every time.

At this point, Underpass Choreographer already had deterministic output contracts and deterministic rejection (NoValidProposal), configurable from the ceremony YAML.

5. The Three Profiles

The comparison is about governance, not models: what changes is the rule for accepting an output.

Five real review tasks, all design decisions about Choreographer itself. Three profiles. 15 canonical executions - one for each task x profile cell. The full experiment left 25 recorded executions: the 15 canonical ones in the table, 8 ceremony failures (4 rejected by the rule-based judge and 4 caused by operational issues around the runner, the script that launched ceremonies), and 2 duplicate reruns. Everything is preserved under experiment/runs/.

The main metric, unsupported_claim_escape_count, answers one concrete question: how many claims reached the final decision without an existing reference in the evidence pack?

Important precision: in these runs, "existing reference" does not mean "proven claim". The rule-based judge checks that the citation exists in the pack. It does not evaluate whether the citation supports the sentence. That second layer is already under construction in Choreographer, but it was not part of the experiment measured here.

6. What I Measured

This is the controlled experiment I ran on July 4, 2026, after the initial failure. These are not handwritten notes: every row can be reconstructed from persisted artifacts with a trace_id.

Profile A, with no contract, did exactly what I feared: all 14 of its claims reached the decision without a single evidence reference. In the profiles with a contract, none of the 36 claims reached the decision without an existing reference in the evidence pack. That does not automatically make them true. It only removes the sentence that cannot even point to where it came from.

The most useful contrast appeared in task 1. With the same question and the same evidence pack, B closed as accept and C closed as reject. In C, the path is visible: the initial proposal had two claims, the reviewer on the L4 marked one BLOCKING objection, and the final version came out with one claim. The trace lets you verify those three steps later, even after the worker that ran the reviewer is gone.

The infrastructure side looked like this:

• Cold start: 77 seconds until the instance was running; 306 seconds until the reviewer could serve inference. In other words, about five minutes before the L4 was actually useful. That includes joining the cluster, container startup, vLLM, and loading weights.
• Once warm, I did not pay that startup cost again while the worker stayed alive: the 5 profile C decisions, each with 3 steps, took 92 seconds of deliberation (104 seconds including runner bookkeeping between decisions; sources in experiment/RESULTS.md). I did not measure per-request latency. I measured the operational difference between keeping the worker warm for a few minutes and letting it shut down.
• Autonomous shutdown: the reaper stopped the worker at 13:23:11 after 350s of measured idleness. The first real line in its events.log was: idle_shutdown instances=i-084f... idle_for=350s.
• Total worker lifetime: 15.5 GPU-minutes - about $0.22 at the on-demand price for g6.xlarge in eu-south-2 ($0.848/hour in the AWS Price List at the time of writing, July 2026; EBS and data transfer separate).

The worker kept nothing. What survived lived outside it: ~32 KB across profile C decision artifacts and responses, plus traces and logs in the observability layer. That was enough to reconstruct what happened without starting the instance again.

7. Reconstructing a Decision After the Worker Shut Down

I wrote a Rust terminal tool, decision-trace-tui, whose entry point is not the cluster. It starts from the decision artifact: a persisted JSON file with the trace_id, the profile, the claims, and the worker lifecycle. From that identifier it goes to Tempo for spans, Loki for deliberation events, and the worker state to check whether the machine is still alive.

This is the case I cared about: task 1, profile C (evidence_bound_review), using the evidence from the initial failure. When I opened the trace, the worker had already been shut down by the reaper:

== decision exp-task-1-c-r1-131151 (evidence_bound_review) ==
status: reject - trace: 80157a56... (6.9s, 6 spans)
claims: 1 total, 1 with evidence, 0 ESCAPED
worker: g6.xlarge i-084f... idle_shutdown=true alive_now=Some(false)
-- spans --
rpc.run_ceremony (6933.7ms)
  run_ceremony (6933.3ms)
    deliberate (1604.4ms)   <- primary proposes (local)
    deliberate (3989.1ms)   <- skeptic objects (from the L4)
    deliberate (1339.6ms)   <- final review against the contract
-- events (33) --
11:11:52 validator=claims-evidence-grounded passed=true
         verdict=all 2 claims grounded in the evidence pack (5 allowed refs)
11:11:56 proposal drafted author=agent-skeptic_c-0
...

The instance is gone. But the full sequence is still there: who proposed what, what the reviewer objected to, what the rule-based judge validated, and how the decision ended.

When compute disappears, observability stops being debugging. It becomes the decision record.

8. Task 1, From the Inside

The table in section 6 is the main result: in the profiles with a contract, 0 of 36 claims without an existing reference reached a final decision; in profile A, with no contract, all 14 of 14 got through. To keep that from feeling like an abstract table, here is one of those decisions from the inside.

The reviewer blocks a claim. In task 1 of profile C, already inside the evidence-bound experiment, the initial proposal had two claims. The reviewer running on the L4 answered:

OBJECTION to "ev-t1-csi-fabrication" - the term 'privileged:true in the pod spec' is mentioned without context; may require further clarification. [...] BLOCKING overall. Non-blocking decisions are too risky with partial evidence; we seek definitive proof before making a non-rejection motion.

Translated into the mechanics of the run: the reviewer was not merely commenting on the answer. It marked a blocking objection to one of the two claims. In the final step, that claim was removed: the decision ended as reject with a single claim.

The actual decision artifact (task 1, profile C, as stored in S3):

{
  "decision_id": "exp-task-1-c-r1-131151",
  "trace_id": "80157a56cf6cc30994585d31b1cc1a6f",
  "profile": "evidence_bound_review",
  "status": "reject",
  "final_claims_total": 1,
  "claims_with_evidence": 1,
  "unsupported_claim_escape_count": 0,
  "worker_lifecycle": {
    "instance_family": "g6.xlarge",
    "ephemeral": true,
    "idle_shutdown_triggered": true,
    "stopped_at": "2026-07-04T13:23:11+02:00",
    "idle_shutdown_evidence": "gpu-idle-reaper events.log: idle_for=350s"
  },
  "observability": {
    "otel_trace_exported": true,
    "logs_exported": true,
    "metrics_exported": true
  }
}

Honest limitations: this is not a general reasoning benchmark. It is an operational architecture experiment. It has 15 canonical executions over 5 tasks, plus 8 ceremony failures and 2 duplicate reruns preserved as artifacts. The results describe this system, not LLMs in general.

The most important limitation is still the one from section 5: the rule-based judge validated existing citations, not semantic support. The next layer - checking whether the cited evidence actually supports the claim - is already under construction in Choreographer, but it was not part of these runs.

Another limitation: the current rule-based judge rejects whole proposals, not individual claims. That is why rejected_claims is zero in every profile. A first-class request_more_evidence transition was not part of these runs.

9. What the Experiment Leaves Me With

The conclusion is not that decisions need more agents. It is that, if a decision is going to matter later, it needs three things: an output contract, persistent evidence, and a trace that lets you reconstruct how it got there.

The number of models is secondary. What matters is what can happen after the model stops talking.

The repo is not an appendix; it is the reproducible part of the experiment. It includes the Terraform code, Helm charts, operational scripts, and artifacts behind these numbers: underpass-ai/aws-side-cluster.