# Should a builder improve the agent harness before changing models?

Canonical: https://mudpie.ai/gtm/should-a-builder-improve-the-agent-harness-before-changing-models/
Breadcrumb: [Home](https://mudpie.ai/) / [Go-to-market](https://mudpie.ai/gtm/) / [Should a builder improve the agent harness before changing models?](https://mudpie.ai/gtm/should-a-builder-improve-the-agent-harness-before-changing-models/)
Author: Ali Abouelatta (https://mudpie.ai/authors/ali-abouelatta/)
Published: 2026-09-19
Updated: 2026-09-19
Research type: article
Method: Timestamped YC Paper Club and Jeff Dean lessons translated into a concrete agent-failure decision table.

Change the harness before changing the model—unless the whole system is already clean and the model still cannot do the job.

That sounds obvious. In practice, model upgrades are attractive because they are easy to name. Context, state, permissions, tool design, and evaluation are harder to see. They are also where a large share of agent failures live.

## The model is one component

Jeff Dean makes this point directly in [The 1% Rule for Building in AI](https://www.youtube.com/watch?v=CxXgV54KzpQ). A useful system may need retrieval, tools, memory, decomposition, multiple approaches, and evaluators around the model. His advice is not “never change the model.” It is to stop treating the model as the product.

The YC Paper Club discussion on [why the harness matters more than the model](https://www.youtube.com/watch?v=n9xKblqyQ28) gets more operational. A long-running agent needs turn, tool, skill, sub-agent, and session budgets. It needs a way to compile context, preserve state, trace actions, and respect permissions. For research work, the harness can define the purpose, seed ideas, evaluator, role-specific agents, review step, and final write or freeze phase.

That is not scaffolding. It is the workflow.

## Diagnose the failure before you buy a new model

| What is failing | First change to try | Why |
| --- | --- | --- |
| The agent does not know the relevant facts | Retrieval, context assembly, or source selection | A stronger model cannot answer from context it never received |
| It has the facts but chooses the wrong action | Tool contract, routing, or approval boundary | The problem is decision structure, not raw language ability |
| It starts well and loses the thread | State, memory, compaction, or session management | Long tasks fail when the system forgets what matters |
| It repeats a known mistake | Trace, evaluator, skill, or regression case | A model swap may hide the symptom without fixing the loop |
| It fails a simple, well-specified task with clean context and tools | Model capability or task scope | This is the case where a model change has earned its place |

The first four are harness problems until proven otherwise. That does not mean they are easy. It means they are local enough to investigate.

## What the Paper Club example changes

The useful part of the talk is the control plane. A research agent is not just a prompt pointed at a frontier model. It is a sequence with roles, budgets, review, and a point where someone decides whether the result can be written.

That gives a founder a better first experiment. Keep the model fixed. Keep the task fixed. Change one system layer.

Maybe the agent gets a clean source bundle instead of a folder of mixed files. Maybe the research sub-agent has a bounded job instead of permission to wander. Maybe the evaluator checks citations before the writer sees the draft. Maybe the final write step is separated from the exploratory steps so a bad intermediate result cannot silently become published output.

The point is not to build a giant harness before a product exists. It is to make the failure legible enough that the next decision is not guesswork.

## When the model really is the problem

Jeff Dean gives a useful opportunity filter: distinguish a task the frontier model performs imperfectly from one it fails almost completely. The first may be a temporary gap. The second may leave room for private data, a specialized model, or a different product surface.

So change the model when:

- the same task fails across clean context, tools, state, and permissions;
- an alternative model succeeds on the same fixture, not just a benchmark screenshot;
- the quality difference survives the workflow's actual latency and cost limits; and
- the new model does not create a worse safety or review problem.

Do not change the model because the output felt slightly dull. That is not a diagnosis.

## The founder decision

I would keep a small failure matrix for the real task. One row per failure, one suspected layer, one bounded change, and one regression case. If the same error disappears after better context or a clearer tool contract, you have learned something about the product. If every clean version fails, then the model—or the task—is the next question.

Short version: the model is the engine. The harness is the vehicle, route, brakes, dashboard, and mechanic. A faster engine does not fix a car that cannot reach the road.


## Author disclosure

I cofound Lazyweb and publish Mudpie. This is an owner-written publication, not an independent testing organization. Research notes distinguish observations, sourced reporting and editorial judgment.