My last post made a bet: as AI gets good at generating software, the durable artifact stops being code and becomes contract-and-evaluation, i.e. the intent a system must honor (its functional requirements, quality attributes, constraints, and compliance obligations) plus the evidence that the intent is actually realized. The code itself demotes to a swappable implementation detail.

The post drew a lot of agreement. But the most valuable response wasn't agreement. It was a precise, uncomfortable challenge, and it landed squarely on the weaker half of the pair. Not the contract. The evaluation. This post is an attempt to take that challenge seriously, because I think it's right and because getting it wrong is how the whole idea fails.

Here is the objection in its strongest form: For fifty years, the way we derived running systems from higher-level descriptions was deterministic. A compiler turns source into machine code; a model-driven transformation turns a model into an implementation. If the transformation is correct, the output is correct; by construction. You don't have to test every compiled program to believe the compiler preserved its meaning. Correctness flows from the derivation itself.

Generative regeneration throws that away. When an AI realizes a contract, there is no meaning-preserving transformation underneath. The output is trustworthy only insofar as you can check it. We have traded conformance by construction for conformance by evaluation. A proof for a measurement. And measurements are weaker than proofs. That is not a detail. It is the central cost of the approach I proposed and I glossed over it.

It gets sharper. A test result is not a free-floating fact about a system. It is indexed to the harness that produced it and to the model that generated the code. Change the generator and you change the distribution of mistakes it makes. The evaluation you designed was implicitly ruling out the failure modes of a particular model at a particular time. Swap in a new model, which is  a routine event given how fast models evolve, and the error distribution shifts underneath you. The old evidence was answering a question about a system that no longer exists.

So a green evaluation suite is not a durable guarantee. It is a statement of the form: this contract held, under this harness, against this model, at this moment. Push the horizon out even a few months and the harness itself stops being infrastructure you can trust blindly and becomes an object you have to investigate. Evidence, it turns out, is not a stock you accumulate and bank. It is perishable.

One commenter gave the resulting condition a name I want to borrow, because it is exactly right: knowledge debt. Technical debt is code you understand but that is messy. You know why it works and you know it needs cleaning up. Knowledge debt is the inverse. The code is clean, generated, and passing its checks, but nobody knows why it works, or under which validation regime it was ever shown to work. The system runs. The understanding is gone.

This is the characteristic debt of the regeneration era and the naive version of my own thesis maximizes it. "Don't worry about the code, just regenerate it when you need it" sounds liberating. Practiced without discipline, it means continuously producing artifacts that no human comprehends, validated by evidence whose expiry date nobody tracked. You don't eliminate the maintenance burden. You move it from the code, where at least it was visible, into the assurance, where it is invisible until it fails.

If contract-and-evaluation is going to hold up, it has to answer this. So here is the constructive half. The mistake hidden in the objection, and in my original framing for that matter, is treating evaluation as a gate: a one-time acceptance test the regenerated system passes on its way into production. If evidence is perishable, a gate is precisely the wrong shape. A gate issues a certificate, and the certificate silently expires.

The reframe is the core of the work I have been doing on continuous, trustworthy autonomy: software engineering is migrating its center of gravity from design-time to runtime. A quality attribute stops being a fixed guarantee frozen into structure at design time and becomes a managed variable held by a closed loop: specify what "good" means, observe the current level, detect the gap, and act to close it. Design-time produces artifacts; runtime produces controllers.

Apply that to evaluation and the perishability problem changes character. Evaluation is not something you do once to admit a system; it is a standing, runtime mechanism that continuously re-checks the contract as models, inputs, and the world drift. A design-time acceptance test hands you a guarantee that decays the moment anything changes. A runtime evaluation loop keeps the contract honored as things change, and tells you the moment it can no longer do so. The objection assumed evaluation was static. Once evaluation is a control loop, "the evidence expired" stops being a silent catastrophe and becomes a signal the loop is built to catch.

Abstractions are easy to nod along to, so make it concrete. Suppose one responsibility in a larger system is a component that extracts structured line items from supplier invoices, such as quantities, prices, tax codes, and feeds them into a financial system. Classic candidate for regeneration: well-scoped, useful, not glamorous.

The contract says: extract line items with ≥99% field-level accuracy; never emit a tax code outside the approved set; flag anything ambiguous for human review rather than guessing. The evaluation is a labelled corpus of invoices plus property checks on the output. Now watch what a naive harness misses and a real one has to do.

Version 1. An AI generates the component. It passes at 99.4% accuracy on the corpus. Green. Ship it. A naive harness stamps nothing and moves on. And quietly opens a line of knowledge debt.

A real harness stamps the evidence. The passing result is recorded together with the model that generated the code, the harness version, the corpus, and the accuracy achieved. The claim on file is not "this works." It is "this achieved 99.4% under model M1 and harness H1 against corpus C3." Versioned evidence.

The regime is made explicit. The corpus was almost entirely single-currency European invoices. The recorded claim therefore includes its bounds: validated for single-currency EU invoices; multi-currency and non-EU tax regimes are outside the tested regime. Regime awareness: the harness knows where its evidence stops applying, and the running system can detect when an incoming invoice falls outside it and route it to review rather than trusting a number nobody verified.

Version 2. Three months later the underlying model is upgraded and the component is regenerated; same contract. It still passes overall at 99.3%. A gate would wave it through. But because evidence is model-indexed, the harness re-runs and compares distributions, and finds the new model has shifted its errors: it is now slightly better on typography but noticeably worse on multi-line descriptions that wrap across rows; a failure mode the old model didn't have. Aggregate accuracy hid it; the evaluation caught it because it was co-evolving with the generator, not trusting the old green light. Changing the model triggered re-evaluation instead of inheriting stale evidence.

The value judgment stays human. Finally, the loop reaches a question it is forbidden to answer on its own: is 99.3% good enough for this jurisdiction's audit requirements? That is not a measurement. It is a judgment about acceptable risk, and it routes to a human who owns it. This is the authority axis of trustworthy autonomy: some conflicts resolve by mechanism, some are value judgments the loop must not make, and knowing which is which, and routing accordingly, is the engineering job. It is the structural reason humans remain non-optional and it is not sentimental.

Four properties, then, separate a real contract-and-evaluation harness from the naive version that manufactures knowledge debt: evidence is versioned (stamped with the model, harness, and regime that produced it), contract and evaluation and generator co-evolve (a model change forces re-evaluation, never silent trust), the harness is regime-aware (it knows the bounds where its evidence holds and detects departures from them), and value judgments route to humans rather than being silently mechanized.

This connects directly to the question I left open last time: how large, how critical, and how complex must a system be before an AI can't reliably regenerate it from scratch?

Knowledge debt reframes that question with unexpected precision. For a small, low-stakes, easily-checked component, perishable evidence barely matters. Regenerate freely and the cost of a miss is trivial. For a large, critical, complex system, the same casual regeneration is how you end up operating something nobody understands, whose assurance quietly expired. The frontier of what is reliably regenerable turns out to coincide with the frontier of what is affordably evaluable. The threshold isn't really about whether the AI can write the code. It's about whether you can build an evaluation loop good enough to keep trusting the code it writes.

Eliminating code was never the hard part; models can already do more of that than most organizations have admitted to themselves. Making evidence durable, that is versioned, regime-bounded, and co-evolving with the generators that keep changing underneath it, is the hard part, and it is largely unsolved. That is where I think the interesting work of the next few years actually sits, and it is a large part of what my own research and the work in Software Center is turning toward. For example, we are kicking off a large project on Continuous Compliance (contact me in case you’re interested to learn more).

It is also, fittingly, one more instance of the pattern that runs through everything I am about to write about this fall: value quietly relocating. Last time it moved from the code to the intent. This time it moves again — from the artifact to the assurance that the artifact behaves. Next week the series proper begins, at the bottom of the stack, with the compute substrate everything else is built on.

Keep the objections coming. The good ones are worth more than the applause. To end with a fellow Dutchman: Dijkstra warned us in 1970: "Program testing can be used to show the presence of bugs, but never to show their absence." He was right, and regeneration doesn't repeal it. It makes it the central problem. If evaluation is all we have left, then making that evaluation trustworthy, durable, and honest about its own limits is the discipline the next decade has to build.

Want to read more like this? Sign up for my newsletter at jan@janbosch.com or follow me on janbosch.com/blog, LinkedIn (linkedin.com/in/janbosch) or X (@JanBosch).