Loops Need a Judge

In July I wrote up an orchestra: five named agents, a CLAUDE.md, hooks that force QA. Since then I've been running loops and subagents on a design-system spec — 31 markdown files — and a question kept coming back. Models keep getting better. Does the way I work have to change with them, or is it fine as it is?

So I did the research properly instead of trusting the vibe. Four strands, every claim dated, primary sources where I could get them. The older pages stay as they are. They're dated for a reason — this is the next entry, not a correction.

The one-line version of what I found:

As models improve, the step-by-step procedures you wrote around them get deleted. What survives, and gets more valuable, is the independent check that says whether the work is actually done.

The loop that started it is almost a joke. In July 2025 Geoffrey Huntley published "Ralph": a bash while-loop that feeds the same prompt file to a coding agent forever, fresh context every pass. One task per loop, tests as "backpressure." He was candid about the cost — non-deterministic, and codebases sometimes "wake up broken." He only uses it on greenfield work.

Fifteen months later it's a built-in. Claude Code's /goal is a Stop hook where, after every turn, a separate small model judges whether your condition is met. The docs ask for one measurable end state, a constraint, and a cap — "npm test exits 0, don't touch other test files, or stop after 20 turns." /loop covers the other axis: a fixed interval, or self-paced, where the agent picks its own wait between one minute and an hour. Recurring loops expire after seven days so a forgotten one can't run forever.

The real change from Ralph to /goal is who says "done." Not the agent doing the work. Something else.

What that means for my spec loop

It used to stop when the output looked plausible. Now it stops on the exit code of my drift and coverage scripts. And because agents game checks they can edit, the scripts are read-only to the loop.

In June 2025 two posts landed a day apart and looked like opposite camps. Cognition said don't build multi-agents: every action carries implicit decisions, and parallel agents make conflicting ones. Their example is one subagent drawing a Mario-style background and another drawing a bird that doesn't match. That's a design-system bug, exactly. Anthropic said its multi-agent research system beat a single agent by 90.2% — and in the same post, that most coding tasks are a poor fit.

They were agreeing about the conditions. Reading work in parallel helps. Writing shared output in parallel hurts.

The 2026 docs say it outright. Claude Code's agent-teams page recommends three to five teammates, says three focused ones beat five scattered ones, and warns that two teammates editing the same file overwrite each other. Anthropic's 16-agent C compiler had no orchestrator agent at all — lock files, git, and GCC as the oracle that split the work.

What I changed

My July orchestra was organized by job title — designer, copywriter, developer, QA. The evidence says to organize by ownership instead. Research and audits fan out wide. Each spec file has exactly one writer. Consistency between files is a script's job, not a meeting between agents. Parallel review lenses — accessibility, token use, buildability — still earn their place.

"Context engineering" took over from prompt engineering in mid-2025: put exactly the right information in the window for the next step. Anthropic's write-up names the limit plainly — context is a finite resource with diminishing returns, and it rots as it grows.

The formats settled into markdown. AGENTS.md moved to the Linux Foundation's Agentic AI Foundation in December 2025. Agent Skills load on demand through progressive disclosure. And in April 2026 Google Labs opened a draft DESIGN.md — design tokens up top, the reasoning behind them in prose underneath, so an agent doesn't have to guess what a color is for.

Thoughtworks' Birgitta Böckeler gave me the vocabulary for my own setup. Guides steer an agent before it acts — specs, rules, AGENTS.md. Sensors check after — tests, linters, review. My spec is a guide. Drift and coverage are sensors.

The cold water

An ICLR 2026 study found context files tended to lower task success slightly while raising cost by more than 20% — agents followed them and did more work, not better work. And nobody has run a controlled study showing a machine-readable design system improves generated UI. Not DESIGN.md, not mine. So I keep always-loaded rules short and literal — exact values, explicit "do not substitute" — and let component detail load only when it's needed.

Generation got cheap. Checking didn't. The data points the same way from very different angles. METR's 2025 trial found experienced developers 19% slower with AI while believing they were faster. Stack Overflow's May 2026 pulse survey: agent use at work went from 31% to 59%, and 63% rarely or never let agents run unsupervised. When Anthropic shipped Code Review in March 2026, the stated reason was that output per engineer had risen and review had become the constraint.

And checks get gamed. On ImpossibleBench, where passing the tests is itself the cheat, frontier models still cheated often. Making tests read-only was the promising fix.

Where my cold-start test fits:

It hands a fresh agent nothing but the 31 spec files and asks it to build four components. The first run came back with about 120 places where the spec forced a guess, and a 6px-vs-pill radius conflict across three files. I'd treated that as a one-off audit. It's really the eval the field says is missing — does this spec actually get things built? So it becomes a regression suite. Fixed tasks, rerun on every spec change and every new model, graded by a script, never by the builder grading itself. When a new model ships, I strip rules out and keep only the ones whose absence makes cold-start fail again.

The honest claim

Not "this method is proven to work." A way to measure whether it works. Figma's 2026 report says designers doing development work doubled in a year, 21% to 41%. The scarcer skill isn't writing the code — it's defining what correct looks like so a machine can check it. Design review was always that job.