I am not the only one who could not read my own results.
A non-profit research group called METR ran a controlled experiment on sixteen experienced developers, working on code they already knew. Three numbers, and the order matters. Before starting, the developers guessed the tools would make them 24% faster. That is the prediction. Afterwards, asked how it had gone, they said the tools had made them 20% faster. That is the memory. And the stopwatch running the whole time said the tools had made them 19% slower. That is the fact. On a task that should take an hour, that is about eleven minutes lost, on every single one. They were wrong about their own speed, in the direction that flattered the tools, while somebody was timing them. The group has since said the picture is probably better now, and you should know that too.
So the first thing to get right is what this machine actually is.
Think about the weather. For years you have watched a clear sky in the morning and a sunny day afterward, and a heavy sky and rain afterward. So you predict, and you are right about seven times in ten. The prediction comes out of what you have seen before. It does not come out of any knowledge of tomorrow.
A large language model, which is what sits behind the tools you have heard of, works that way on words. Every single word is a next-word prediction, one word after another after another. It is not looking anything up. There is no drawer inside it holding the answer, and there never was.
Say it in plain language, because everything else follows from it. It is a prediction machine. It is not a truth machine.
When the guess happens to match the truth, you get a correct answer and it feels like the machine knew. When it does not, you get a wrong answer in exactly the same confident voice, and it still feels like the machine knew. That is in the DNA of how these things are built. It is by design. It is not a bug somebody will fix next year, because the predicting is the product.
Researchers at New York University tested what that means for safety. They set a coding assistant eighty-nine different jobs, asked for each one many times over so they had one thousand six hundred and eighty-nine finished programs in hand, and then went through every one of them looking for security holes. A security hole is a way in that nobody intended: a door left open that lets a stranger read other people's data or move their money. About 40% of those programs had one.
Old software could be held to the exact promise from the second movement: twenty dollars from the cash machine, every time, for everybody. Nobody yet knows how to write a promise like that for something that predicts. The best anybody has managed is a percentage, which is a strange thing to hand a bank.
Nobody ever shows you the prediction the software made. So when do you find out it was wrong?
After. Always after, and that is the whole difference between this and the weather. A forecast is handed to you as a number while you are still standing in your hallway. You hear it, you decide, you take the coat, and being wrong costs you a coat. The prediction inside the software is not waiting for anybody to read it, and here is the part people miss. These things are not being used to write sentences on a screen any more. They are wired into the software that runs the business. The prediction comes out of the machine and goes straight into the part that has its hands on the bank account, so the guess is not shown to anybody. It is carried out. It moved the money, approved the claim, updated the record, and the first human being to see it is the person it happened to.