Loop Engineering — Simplified.

Last Updated on July 27, 2026 by Editorial Team Author(s): Darshandagaa Originally published on Towards AI. loop engineering “My job is to write loops.” That’s Boris Cherny, who leads Claude Code at Anthropic. He’s said he stopped prompting Claude directly and now spends his time designing the loops that prompt it for him [1]. That line, and a couple of others like it, kicked off a wave of loop-engineering explainers this year [1][2]. I read six of them. Then I built one. Two pieces, specifically — the two that every explainer mentions and almost none actually run. Run-until-done: feed the model its own real test failures instead of asking it to guess again. Maker/checker: don’t let the model that wrote the code decide whether the code is correct. I built both from scratch, about 600 lines of Python, wired to claude-opus-4-8, graded against MBPP+ [3]. Total spend across every experiment in this article: under two dollars. And the second piece — the one every write-up treats as the safe half, because it "actually runs tests" instead of just trusting the model's word — did something in my own numbers that none of those explainers warned me about. TL;DR: Loop engineering’s two core pieces are simple to wire up and easy to get quietly wrong. My “real feedback” loop looked identical to random retries until I found the bug in my own test harness. My “safe” test-running verifier had a higher false-accept rate than a checker that just asked the model how confident it felt. Building the loop is the easy 20%. Wiring a Loop to Nothing Most of what gets published about loop engineering stops at the wiring diagram. Trigger, verifiable goal, tools, state, stop rules — five boxes, one arrow between each, done. The implication is that once the boxes are connected, the loop works. That’s the same logic as installing a smoke detector and calling the house safe. The detector is on the ceiling. It’s wired in. Nobody checked whether there’s a battery in it. I hit this exact failure with my “real feedback” loop. It was wired to the actual test output, not a generic retry prompt. On paper, it should have clearly beaten a loop fed nothing but “that was wrong, try again.” My first run said otherwise. The Grader That Grades the Grader Before touching the loop, I built the thing everything else depends on: a scorer that runs candidate code in an isolated subprocess with a hard timeout, and grades it against hidden tests. I didn’t trust it until it graded itself. Feed it a known-good solution — it has to pass. Feed it a known-bad one — it has to fail, with the assertion error attached. Feed it an infinite loop — it has to get killed by the timeout, not hang forever. def scorer_selftest() -> None: tests = ["assert add(2, 3) == 5", "assert add(-1, 1) == 0"] good = run_tests("def add(a, b):\n return a + b", tests) assert good["all_pass"] bad = run_tests("def add(a, b):\n return a - b", tests) assert not bad["all_pass"] and "AssertionError" in bad["stderr"] loop = run_tests("def add(a, b):\n while True:\n pass", tests, timeout_s=3) assert loop["timed_out"] Then I validated the whole pipeline against 75 MBPP+ reference solutions. All 75 passed. Only after that did I trust a single number the loop produced. The Loop That Looked Fine and Wasn’t The loop itself is almost insultingly simple. Generate a solution, grade it, and on failure, feed the real stderr back — not “try again,” the actual error — for up to three attempts. I also built a control arm, because I didn’t want to trust a headline number without one: run the identical loop, but replace the real error with a generic “that was wrong, write a different solution.” If real feedback doesn’t clearly beat that, something in the wiring is broken. First run, 35 problems: single-shot (pass@1) 32/35 91.4%loop, real feedback 32/35 91.4% ← identicalloop, generic feedback 32/35 91.4% Identical. All three arms. That’s not a loop working, that’s a red flag wearing a loop’s clothes. I went digging into the failures instead of the headline number, and found it: MBPP+’s hidden-test harness was failing with a bare AssertionError — no failing input, no expected value, no actual value. "Real feedback" was informationally identical to "try again," because there was nothing in it the model could act on. I instrumented the harness to report the failing input, the expected output, and what the code actually returned. Same 35 problems, second run: single-shot (pass@1) 32/35 91.4%loop, real feedback 33/35 94.3%loop, generic feedback 32/35 91.4% Real feedback recovered a problem the generic arm couldn’t touch, for about 2,500 extra input tokens across the run. The loop was never broken. The signal it was wired to was empty, and only the control arm surfaced that — the headline metric never would have. The Verifier That Failed the Way the Theory Didn’t Predict The loop needs a stop rule, and “the model says it’s done” isn’t one. So I built a checker that writes its own tests from the spec — never seeing the hidden tests, never seeing its own solution’s code — and then actually runs them. Accept only on a clean sweep. Default to reject. I compared it against three weaker checkers on 41 candidates my loop had produced, 33 correct and 8 wrong, measuring false-accept rate: how often each checker waves through code that’s actually broken. checker false-accept false-reject trust everything 8/8–100% 0/33–0% ask the model if it’s confident 2/8–25% 4/33–12% a second model reads the code 2/8–25% 5/33–15% writes tests and runs them 3/8–38% 1/33–3% I expected the test-running checker to win outright on false-accepts. It didn’t. It let through a higher fraction of wrong code than either opinion-based checker. The reason mattered more than the number. All 8 wrong candidates came from three problems with genuinely ambiguous specs. The checker and the fixer were the same model reading the same ambiguous sentence — so the checker’s self-written […]