Chapter 8 of 17

Regression is the number one problem with agentic development

One word pays your salary in this field, and it is worth slowing down for.

Think about everything you can do on Amazon. Open an account. Log in. Search for a product. Compare two products. Put one in the basket. Check out. Cancel the order. Return an item. Update your credit card. Change the delivery address. Write a review. Each one of those is a scenario, each has variations, and a company that size has ten thousand of them and more. A bank like Bank of America is the same.

Now somebody changes one line of code this morning.

That is the problem. Nobody knows whether the other ten thousand scenarios still work. Login worked yesterday. Does login work now? Checkout worked an hour ago. Does checkout work an hour from now? Nobody knows, because nothing was touched on purpose, and things break anyway.

Re-running everything that used to work, to prove it still does, is called regression testing. The word regression by itself means the opposite thing: it is when something that used to work has quietly gone backward and stopped working. So the damage is called a regression, and the work you do to catch it is called regression testing, and you will hear people shorten both of them to just regression. That is confusing and I did not invent it. Change one line of code and anything else could break. That is the single biggest risk in building software, and it is why, every time anybody adds anything, the entire set has to be run again.

Now put an agent on it. You can build anything you want, and the feature arrives, and that part genuinely is not the problem any more.

The first question everybody asks is: can you build it? The answer now is yes, always, quickly. So ask the second question instead. What happens when you add code on top of code?

The thing you asked for works. The checks are what nobody looks at. A check here is a written test, built once by a person and kept, so it can be run again whenever anybody wants. A serious company has thousands of them saved up, one for each thing the software is supposed to do. Those are the ones nobody ran, and you do not know which of them broke. Regression is the number one problem with agentic software development. Not building. Breaking what was already built, quietly, while you were busy being impressed.

And the damage a single change can do grew with the volume. Ten thousand scenarios, re-run every week, in every browser people use and on both kinds of phone. Done by hand, that is tens of thousands of hours a year, which is a room of ten people doing nothing else, forever, just to make sure the thing that worked yesterday did not get taken down because an agent inserted one line of code somewhere nobody was looking.

You added one feature this morning and something you shipped last month stopped working. Nobody went near last month's code. So how?

Because none of it is separate. Software is not a shelf of objects standing next to each other. It is one thing wired through itself, and the pieces get shared. The login screen and the invoice both need to know who you are, so somebody wrote that once and both of them use it. There is one copy, not two. Change it so the invoice prints your full name, and the login screen is now looking at the same changed thing, and nobody decided that. Nobody wrote that connection down, which is exactly why nobody thought to check it. And you cannot write them all down: a large application has millions of these shared pieces and the connections between them change every week, so any map of them is out of date before it is finished. That is why the only defense anybody has found is the brute one. Run all of it again, every single time.