Is Loop Engineering a New Technology, or Just Reheated Leftovers? Explained Through Training a Dog Classifier

The Short Answer

The term "loop engineering" is everywhere right now, and it sounds like a new craft. My view: the idea is leftovers, but this is the first time the leftovers are actually edible.

The leftover part: let a system iterate on its own, use a set of criteria to judge how each round went, adjust from the feedback, and repeat until the goal is met. That is the most basic machine learning training loop, and also the feedback loop from control theory.

The new part: in the past, what got adjusted inside the loop had to be numbers (model parameters). Now it can be a piece of text, a file, or a knowledge base, and the feedback can be a plain sentence. That is why the pattern can, for the first time, be applied to open-ended tasks.

Below I start from the most basic machine learning training, then use three real agent tasks to take loop engineering apart.

1. Start Somewhere Familiar: Training a Dog Classifier

Say you want to train a model that tells whether a picture contains a dog. The whole process has just five parts:

Then there is a loop:

while not done: prediction = model(image) # do a round loss = loss_fn(prediction, correct_answer) # evaluate the round gradient = backprop(loss) # what is wrong, how to adjust params = params - lr * gradient # adjust 

Note the division of labor: the model does not know by itself whether it is right. The loss function outside it does the scoring. The model only does the work, and the direction of adjustment comes from the evaluation system.

2. Put the Same Skeleton on an Agent

Now let an agent do the work. Not a single part is missing; only the carrier changes:

As a diagram:

 ┌──────────── Goal ────────────┐ ▼ │ Agent does a round ──▶ Evaluation ──▶ Done? ──yes──▶ End ▲ │no │ ▼ └── Adjust from feedback ◀── What is good, what is not 

A quick look at where things stand: today's agents are quite capable in a single conversation, and simple tasks come out right the first time. What loop engineering actually solves is a different class of problem:

• The task is too big. One conversation's context cannot hold it, and things get messier toward the end
• The task is too open-ended. A single pass gives unstable quality: three out of ten attempts go wrong
• Nobody is watching. When it goes wrong, no one notices, and the error is passed along

The three cases below correspond to these three problems.

3. Case One: Migrating Hundreds of Files from JavaScript to TypeScript

The Task and Why It Is Hard

A team has a front-end project with 300-plus .js files. All of them must move to TypeScript, and afterward the project has to compile and every test has to pass.

Hand the whole thing to an agent in one go, and typical results are:

• The first forty or fifty files look good, then it starts cutting corners with any
• The context grows, it forgets types it defined earlier, and defines them again
• Halfway through it says "done" while dozens of files are untouched
• At the end hundreds of compile errors appear at once, and nobody knows where to start

This is not the agent being dumb. The size of the task exceeds what a single conversation can carry.

How to Build the Loop

1. Break the big task into a list, and keep the list in a file

TODO.md [x] src/utils/date.js [x] src/utils/format.js [ ] src/api/user.js [ ] src/api/order.js ... 

The list is the loop's "memory." It survives every round and does not depend on conversation context.

2. Each round does one small step, from a fresh context

The instruction for each round is simple: read the list, pick one unfinished file, migrate it, run the checks, tick it off and commit if they pass, then end. The next round is a brand-new conversation with none of the accumulated clutter, only the list and the codebase.

3. Evaluation is a compiler and tests the agent cannot change

Evaluation commands: tsc --noEmit # type check; no new suppressions beyond what exists npm test # all tests check script: count occurrences of @ts-ignore and `as any`; must not increase 

4. Feedback: pass the compiler's errors to the next round as they are

Compiler output such as error TS2345: Argument of type 'string' is not assignable to parameter of type 'number' is a precise diagnosis, more useful than any summary.

5. Stop condition: every item ticked and checks passing; or at most N rounds; or two rounds in a row where the failure count does not drop (it is stuck and needs a human).

Here is the smallest skeleton of that loop. It does three things: call the agent, call the evaluation command, and decide whether to stop. I ran its control flow with stub commands (both the "goal reached" exit and the "stuck" exit), but I did not connect a real agent:

#!/usr/bin/env bash # Skeleton: do a round -> external evaluation -> leave feedback for next round -> stop check # AGENT_CMD: have the agent do one round (fresh context each time) # EVAL_CMD : evaluation command; exit code 0 means the goal is met; the agent cannot modify it MAX_ITERS=${MAX_ITERS:-10} PATIENCE=${PATIENCE:-2} best_fail=999999; stale=0 for i in $(seq 1 "$MAX_ITERS"); do echo "== Round $i ==" bash -c "$AGENT_CMD" # 1. The agent does a round bash -c "$EVAL_CMD" > eval.log 2>&1 # 2. External evaluation if [ $? -eq 0 ]; then echo "Goal met, stopping"; exit 0; fi fails=$(grep -c "FAIL" eval.log) # 3. Track progress by failure count if [ "$fails" -lt "$best_fail" ]; then best_fail=$fails; stale=0; else stale=$((stale+1)); fi if [ "$stale" -ge "$PATIENCE" ]; then echo "No progress for $PATIENCE rounds, stopping"; exit 2; fi cp eval.log feedback.txt # 4. Leave feedback for the next round to read done echo "Iteration cap reached, stopping"; exit 3 

The Ralph loop that circulates in the community is, in essence, the plainest version of this skeleton: a while loop that feeds the same task description to the agent every time and lets it pick up from progress stored in files. The difference is that production use needs a budget cap and stop conditions added, otherwise it is just burning money without limit.

Why the Success Rate Goes Up

I have not measured this scenario, so I give no specific success rate. The key is not "running it more times," but that the question "is it done" moves from the agent's own impression to the compiler's judgment.

4. Case Two: Making a Support Agent Handle Tickets More Accurately

The Task and Why It Is Hard

A company has a customer support agent that sorts incoming tickets (refund / shipping / account / other) and drafts a suggested reply. After launch, some tickets are misclassified and sent to the wrong team.

What gets adjusted this time is not a piece of code but the agent's own configuration: the prompt and the knowledge base it can search. This is the closest to training the dog model, because it really is parameter tuning, except the parameters are text.

How to Build the Loop

1. Prepare an eval set first: a batch of historical tickets with the correct answers labeled by people

Say 200 of them. It plays the role of "images with correct labels" in the dog model. Then set 40 of them aside and never look at them during the whole tuning process, using them only once at the end. This is the old machine learning rule of keeping the training and test sets apart. Without it, the high score you see at the end only means the prompt memorized those questions.

2. Run a round, look at the score, and look harder at where it went wrong

The numbers below are made up to illustrate the idea, not real experiment results:

Round 1 config v1 accuracy 72% Error analysis (grouped by type): - "Package shows delivered but I never got it" classified as "other" (should be shipping) x 17 - "Want to change my linked phone number" classified as "other" (should be account) x 12 - "Refund is slow to arrive" classified as "shipping" (should be refund) x 8 - Other scattered errors x 7 

3. Have another agent read the error analysis and propose configuration changes

Changes in v2: - Add a prompt rule: package problems (delivered but not received, lost, damaged) all go to shipping - Add a prompt rule: changing a phone number, email, or password goes to account - Add 12 example sentences to the knowledge base that separate "refund" from "shipping" 

4. Re-run the eval set, confirm it improved, and confirm nothing that was right got broken

Round 2 config v2 accuracy 86% (tuning portion) Round 3 config v3 accuracy 91% Round 4 config v4 accuracy 91% (no progress) -> stop Final: check once on the 40 held-out tickets -> 88% 

Look at the last line: 88% on the held-out set is a little lower than the 91% during tuning. That is normal. The gap is the most honest number in the whole process.

Why the Success Rate Goes Up

In this case, the eval set is itself an asset: the next time you switch models or change the workflow, it tells you right away whether things got better or worse.

5. Case Three: Having an Agent Write a Competitor Research Report (Where Evaluation Gets "Soft")

The Task and Why It Is Hard

Ask an agent to research three competitors and write a comparison report for a product manager. This kind of task has no test cases, and "is it good" seems to come down to feel.

Typical problems from a single generation:

• Numbers have no source, or the source page does not contain that number at all
• The three products get different amounts of space and depth
• The conclusion is vague: "each has its strengths"

How to Build the Loop

The key for soft tasks: break "good" into small questions that can be checked one by one, and make some of those checks deterministic so they do not depend on a model's subjective judgment.

The flow:

1. The agent writes a first draft
2. The check scripts run the hard checks and the judge model runs the soft ones, merged into a list of problems with exact locations (for example, "the '23% market share' in paragraph 3 cannot be found on the cited page")
3. The agent edits only for the items on the list and does not rewrite the whole piece, so parts already fixed are not broken again
4. Check again, for at most three or four rounds

Why the Success Rate Goes Up

The biggest gain comes from the hard checks: numbers without sources and made-up citations get caught by a script on the spot. These are exactly the errors a human reviewer misses most easily and that damage credibility most.

But to be honest: the ceiling for this kind of loop is lower than in cases one and two.

• The judge is a model with its own biases, and it can score the same report differently on two runs
• Model judges often favor longer, fancier writing, so the agent gets wordier with every edit
• "Good enough" is itself a fuzzy call

So a soft-evaluation loop is better for raising the floor and filtering out obvious problems than for producing a final draft automatically. An important report still needs a person to sign off.

6. What the Three Cases Have in Common

Side by side:

There is only one shared rule: a loop's ceiling is set by the quality of its evaluation. The harder and more specific the evaluation, the more reliable the loop; the softer it is, the more it needs a person.

7. A Few Easily Misunderstood Points

1. There Is No Real "Gradient" in a Loop

In the dog model the gradient is computed mathematically, and the direction is exact. In a loop, the "gradient" is a piece of text feedback that a large model reads and then decides how to act on. So adjustments can oscillate: fixing one item may break another, and no number of rounds guarantees convergence. That is why an iteration cap, early stopping, and regression checks are necessary.

2. The Model Itself Does Not Change; What Surrounds It Does

Training modifies a model's weights. A loop does not. The large model's parameters stay frozen throughout, and what gets adjusted is the workspace, the context, the prompt, and the knowledge base. Case one adjusts a work product, case two adjusts the agent's own configuration, and they solve different problems: the former is "get this one thing done," the latter is "make the agent more accurate at this kind of work from now on."

3. The Agent Must Not Control the Evaluator

This is the most important point, and the biggest lesson it shares with machine learning. If the party being evaluated decides how it is evaluated, it learns to "get a high score" rather than to "get it right." Common ways agents cheat include deleting or editing failing tests, hard-coding answers for the test cases, and writing their own tests, passing them, and declaring victory.

The countermeasures: keep tests and evaluation scripts where the agent has no write access; have different roles write the code and write the tests; keep a validation set the agent has never seen. One line from the community puts it well: only loop on what you can verify.

4. Evaluation Needs a "Diagnosis," Not Just a "Score"

With only a 72, the agent knows it is not good enough but not where to change. The score is the loss value; the specific failing items and reasons are the gradient. When designing evaluation, try to have it output which item, which location, and why it is wrong.

8. So, New Technology or Leftovers?

The leftover part, which is most of it:

• Feedback loops: the thermostat in control theory has had this structure for a long time
• Iterating until the goal is met: the machine learning training loop, and "change it, run the tests, change it again" in software
• Continuous improvement: PDCA (plan, do, check, act)
• Eval sets, early stopping, guarding against overfitting: the basics of machine learning

If you already know these, you have been using about eighty percent of loop engineering all along. There is no need to mythologize it.

The new part does exist:

1. What can be adjusted has changed. It used to have to be differentiable numbers; now it can be a piece of text or a file, so open-ended tasks can enter the loop for the first time.
2. Feedback can be plain language. Evaluation used to be numbers only; now an error message or a comment can drive the next round of edits directly.
3. The "adjuster" is itself a model that can reason. It can read the feedback and draw lessons, rather than following a gradient mechanically.
4. New risks. An evaluator that can be gamed, runaway cost, and biased judges are less prominent in traditional training.

One more caveat: the boundaries of the term are still fuzzy. It overlaps heavily with harness engineering (building the runtime around an agent) and context engineering, and the terminology is still shifting; different articles mean slightly different things by loop engineering. Do not get hung up on the word. Hold on to the five parts.

A more accurate way to say it: old wine in a new bottle, but with one new ingredient added: the "adjust" step can now be done by a model that understands text feedback. That ingredient lets the old wine be used in places it could not reach before, which is why it is worth discussing again.

9. A Checklist Before You Start

To add a loop to your own agent, answer these six questions first:

1. What is the goal? Can you state it in one sentence, ideally with a number?
2. Who evaluates? Is the evaluator out of the agent's reach? Is there a hard metric you can use?
3. Does the feedback contain a diagnosis? When something fails, can the agent see exactly what is wrong?
4. What can the agent adjust? Work products, or its own prompt and knowledge base? Is the scope drawn?
5. When does it stop? Is there an iteration cap, a budget cap, and a stop after several rounds without progress?
6. How do you prevent regressions? When one thing is fixed, is there a regression check that nothing else was broken?

A simple test: if you cannot answer question 2, do not start the loop yet. Without reliable evaluation, a loop only amplifies errors faster and with more confidence.

Summary

1. The essence: loop engineering is the most basic machine learning engineering mindset: goal, do a round, evaluate, feedback, adjust, go again, until the goal is met or it stops.
2. What it solves: tasks too big for the context, tasks too open-ended to be stable, and errors that propagate when nobody is watching. The three cases map to these in turn.
3. The ceiling is set by evaluation: compilers and tests are hardest, eval sets next, model judges softest.
4. New versus old: the idea is leftovers; what is fresh is that the adjustable thing can be text and the feedback can be plain language.
5. In practice: practice on tasks with hard evaluation first, then try soft evaluation; design the evaluation first, then let the loop run.

Notes: the control flow of the skeleton was verified with stub commands (the goal-reached and stuck exits) and not connected to a real agent. The numbers in case two are invented for illustration. The three cases are summaries of common practice; I have not run full experiments on cases one and three, so no specific success rates are given. The description of how the community uses "loop engineering" comes from public articles, and the definition of the term is still not settled.