When rebuilding is cheaper than maintaining, keep the spec and the eval set

If rebuilding costs less than maintaining, the code is the part you throw away. Two things last: a one-page spec that says what goes in, what comes out and what a good result looks like to the user, and an eval set you own that runs every time the model changes. We think a founder should buy those two things first, before choosing who builds. They outlive whoever writes the code.

The shift, from the sellers' side

For years, software vendors answered "we'll build it ourselves" with "fine, but think about maintenance." Over the last two weeks on LinkedIn, Gainsight's CEO wrote that this argument no longer works. The comments under the post took it further. Build vs buy is turning into build vs rebuild, and if rebuilding gets cheaper than maintaining, the spec is worth keeping and the code isn't. The same thread raised the obvious objection: rebuild-with-us sounds like a retainer that never ends. The reply was that the vendor's real value is whatever stays stable between rebuilds.

The agency inbox shows the same thing. Under Upwork's August 2026 hiring report, an agency owner described it this way. The small fixed-price job they used to be invited to ten times a week has mostly disappeared. What replaced it is a six-figure integration, and the client opens by saying they are not sure what they need. The conclusion in that thread: when the scope runs far past the budget, every contractor guesses the client's priorities, and the proposals can't be compared. A one-page spec you can send to three contractors is the work, and it outlives any of them.

What the stable layer is made of

We split a product into two layers.

STABLE (you own it, it survives rebuilds)
  spec            one page per task: inputs / outputs / what good looks like
  eval set        real cases + the command that runs them
  data model      what is stored and what it means
  contracts       the integrations other systems depend on

DISPOSABLE (rebuilt when it is cheaper than patching)
  code
  prompts
  the model snapshot of the month

1. The spec: one task contract per job

A talk titled "Before You Build an AI Agent, Do This First" gives the simplest working form: for every task, write inputs, outputs and what a good result looks like. Without that last line, QA comes down to someone reading the output and thinking it sounds clever. Here is the template we use:

TASK:        <one verb, one object>
INPUTS:      <what arrives, in what shape, from where>
OUTPUTS:     <what leaves, in what shape, to whom>
GOOD LOOKS LIKE (in the user's words):
  - <what the user sees and can do next>
  - <what the user must never see>
OUT OF SCOPE: <what this task does not decide>

The line that matters most is "in the user's words." We learned that from our own product, not from a thread.

What happened on Holt

The first version of the Holt dictionary had 10,826 senses. It passed all 28 acceptance criteria. Its golden gates came in at 98.7% and 96%. When the owner opened it, it took minutes to see that the learner had been handed a thesaurus, not a dictionary: the same word, news, three times over.

Nothing in the criteria was false. They described the system (counts, formats, gate scores) and not what a learner sees on screen. Since then, "good looks like" in our specs is written from the user's side:

BEFORE (described the system)
  - every lemma has >= 1 sense
  - golden gate pass rate >= 95%

AFTER (describes what the learner sees)
  - a learner looking up a word sees distinct meanings,
    not the same word repeated as its own definition

2. The eval set: the part you own

In an r/SaaS thread asking what a client should get when an agency hands over an AI feature, the answers agreed: the eval set matters more than the prompts. The shape they described is specific:

evals/
  cases/
    never_regress/     # must pass, every run
    judgment/          # graded, tracked over time
  model.lock           # pinned model snapshot, not an alias
  run                  # one command -> pass rate + diff vs last run

30-50 cases, drawn from real traffic

There is a reason this has to be engineering and not a checklist. In one practitioners' thread, an LLM judge scored the same 145 cases in three identical runs and disagreed with itself on 14 of them. The rule practitioners use is to run each pinned case about five times on the accepted version, record the spread, and count something as a regression only when it falls outside that spread. Pin the exact model snapshot, not an alias that moves underneath you.

for each case in never_regress + judgment:
    baseline = run(case, accepted_version) x 5   # record the spread
    result   = run(case, candidate)
    flag only if result is outside baseline spread

Why buyers are already asking for this

Under vendor checklists, buyers ask whether the vendor is model-agnostic. The worry behind the question, as one commenter put it, is that every model release will turn into a testing project for their own team. The worry is reasonable. In one team's case, a single prompt edit to a reviewer agent looked fine on the cases it was tuned on, then wrongly blocked 17 of 30 normal requests on a set it hadn't seen. A user in another thread said the model they had built on was retired five months after its release. Model changes are now routine.

This is already reaching contract terms. SFAI Labs' pricing manifesto (May 2026) says a milestone is paid only after the buyer reproduces a named eval threshold in their own environment, on datasets, rubrics and pinned judge versions handed over at kickoff. Re-checking on a new model release is a separate contract line. The eval set has become something the contract is about.

What we think a founder should do

This is our view, not a finding:

• Before you pick a builder, pay for the stable layer: one task contract per job, with "good" written in the user's words, and a starter eval set made from your real cases.
• Send the same spec to everyone you're considering. If the proposals still can't be compared, the spec isn't finished.
• Make the eval set yours: your repository, your pinned model, your command. Whoever builds after that, you'll be able to tell whether they kept what worked.
• Treat a prompt or model change as a release that runs the eval set, not as a tweak.

This also answers the endless-retainer objection. The ongoing work is rebuilding. The thing you pay for once is the yardstick.

What would change our mind: evidence that teams rebuilding with an owned spec and eval set end up with no fewer regressions than teams that keep patching the same code. We haven't seen that yet.

FAQ

Isn't the spec just a requirements document? Not quite. A requirements document describes the system. The task contract describes what the user sees and must never see, and each line of it can become an eval case. Holt's dictionary met every requirement and still failed that test.

How big should the first eval set be? The handover thread suggests 30–50 cases from real traffic, split into "must never regress" and "judgment calls," behind one command that prints a pass rate and a diff.

Can an LLM judge grade it? Yes, but measure the judge first. One judge disagreed with itself on 14 of 145 cases across three identical runs. Run each case several times and flag only what falls outside the spread.

Does this mean we never maintain code? No. It means the decision between patching and rebuilding becomes a cost question you can answer, because the eval set tells you whether either one kept what worked.

Who should own the eval set, us or the vendor? We think you should. It's the one artifact that lets you switch vendors, switch models, or rebuild without starting the definition of "good" from scratch.

Related case

Holt is our language-learning product, and the place where we learned to write "good" in the user's words. The case is at iloblique.com/holt, and you can try the product at holt.il-ob.com.

If you have something worth building, we'd like to hear about it.

Sources

• Gainsight CEO's LinkedIn post on build vs buy and its comment thread (paraphrased)
• Upwork August 2026 hiring report and the agency discussion under it (paraphrased)
• "Before You Build an AI Agent, Do This First", video
• r/SaaS thread on what an AI-feature handoff should include
• Practitioner threads on prompt and model-change regressions and on LLM-judge variance (paraphrased)
• SFAI Labs pricing manifesto, May 2026
• Holt dictionary v1 acceptance results, ilOblique