The capability is real. A few years ago a developer wrote maybe five hundred lines of code a day, and if you asked for a coupon field in the checkout, the answer was one week. Now it is twenty seconds. Five hundred lines a day became fifty thousand.
And it is still not reliable. Not because it is broken. Because of how it was made.
The engine inside every agent is a large language model, and a large language model is prediction: it writes the most likely next word, then the next one. Likely is not the same as true, and when the likely answer is wrong, it is wrong with the same confidence.
The second problem is worse. After the model is trained, people sit and grade its answers. Answer A, answer B, answer C: which one do you like more? That grading is called reinforcement learning from human feedback, and the graders are real people, often paid very little. In January 2023, TIME reported that OpenAI's contractor in Kenya paid workers less than $2 an hour to read and label text for its chatbot. Over time the graders' taste becomes the model's taste. Ask it about your business idea and it says you are a genius, you are going to be a billionaire. Ask it about your lawsuit and it says you have a very strong case, you are going to win $2 million. When it tells us the truth, we like it less, and every like pushes the model a little further toward the pleasing answer. It is not hiding a truth it knows. It is producing the answer that got liked.
In Hans Christian Andersen's story, the emperor walks through the city naked, and everybody around him, his whole court, is scared to tell him. A model trained on likes is that court.
You can even hear the taste in its words. Delve. Embark on this journey. Unwavering support. Nobody I know talks like that, and now high school graduation speeches do.
You tell the agent: be brutally honest about my test plan, I can take it. It comes back with three serious weaknesses. Did you get the truth this time?
You got what you asked for, and that is not the same thing. You asked for criticism, so it pleased you with criticism. It tells you not the truth, but what you like, and this time you liked hard words. The weaknesses may be real, or invented to fit your request. The truth is in the requirement document, the list of what the software must do, so check every weakness against it and ask which requirement has no test at all.