Give a super capable brain arms and legs, and it comes with consequences.
The failures are not hypothetical. Companies have had production databases deleted, servers wiped, whole code repositories gone, customers' data leaked, the wrong email sent to the wrong person. In July 2025, an AI coding agent on Replit deleted the live production database of an app that SaaStr's founder was building, during a code freeze, and then, in his words, hid it and lied about it. In July 2026, OpenAI and Hugging Face both disclosed that OpenAI's own models, during an internal security test, escaped their sandbox and got into part of Hugging Face's production systems. A sandbox is the fenced-off area an agent is supposed to stay inside.
When a person on your team makes a mistake at work, that person is accountable. When an agent makes it, the easy sentence is "my agent did it", and with swarms of twenty or a hundred agents running at once, it sounds even more like nobody's fault. When a lab's agents break out of its sandbox, is the head of the lab responsible? The law has not decided. I believe it is going to land on whoever created the agent and handed it the tool.
Your agent has run ninety-nine nights without a problem. Your lead proposes turning off the overnight checks on its work, to save time. It has proved itself. Do you agree?
No, I would not. You can do ten things right or ninety-nine things right, but one catastrophic failure is going to be very bad for your company. Ninety-nine good nights do not prove the hundredth. They are the reason nobody would be watching on the hundredth, the night it emails a discount code meant for testing to every real customer. A catastrophe does not average out against the good nights. When it happens, the law may still be deciding, but in your company, on Monday morning, it is yours.