They ask: Give me a concrete failure you have seen from an agent doing testing.
You say: The clearest one I use is a car. You ask an agent to check whether the car is good. It comes back and reports four tyres present, paint clean, all doors open. Everything it says is accurate. It never started the engine.
That is the failure mode. Not wrong facts. Correct facts about the wrong things. The agent had no way to know which checks carried the risk, so it ran the ones that were easy to run and reported them all as evidence.
Deciding which checks matter is not something you can delegate to the thing being checked.
The way I put it in class is that the agent counted the tyres. Four of them, all present, correctly reported. It never turned the key.
The answer that ends the conversation early: "you need better prompts." Sometimes. But the deeper answer is that risk judgement is the human's job, and saying so is what separates a tester from somebody who runs a tool.
They ask: How do you verify work an agent says it completed?
You say: I do not accept the summary. I ask for the run.
In one of my own classes I had an agent report a clean pass on a banking application. Every test green. Then I opened the page it had just tested and it returned nothing at all. The feature was broken in the most visible way possible and the report said everything was fine.
So the rule I work by now is that a claim of success is text, generated the same way every other sentence is generated. Evidence is the execution output, the screenshot, the artifact that either exists or does not. If I cannot see the thing that would have failed, I have not verified anything.
The cheapest version of this rule costs nothing. Before you accept a green run, open the thing it tested and look at it with your own eyes. It takes a minute and it has caught more for me than any tooling decision I have made. It is not sophisticated and it does not need to be. The failure it catches is the one where nobody looked.
The answer that ends the conversation early: "we review the agent's summary." You are reviewing a prediction of what a summary would look like.
They ask: If we connect the model to our own documents, does that solve accuracy?
You say: It helps a great deal and it does not close the gap.
Connecting a model to a specific body of company knowledge so it answers from that instead of from general training is called retrieval augmented generation. It is the right move. Answers get much more accurate.
But the model is still predicting. It can still produce something that is not in the documents, in the same confident register as everything else. Air Canada's assistant described a bereavement refund policy the airline did not have. A tribunal held the airline to it, and rejected the argument that the chatbot was a separate party.
That is the shape of the risk. Not an embarrassing answer. A statement made on your behalf that somebody can act on.
So I ask two questions of any grounded system. What does it do when the answer is genuinely not in the documents, and how would anyone notice if it invented one. If there is no answer to the second, the system is not ready for customers.
The answer that ends the conversation early: treating grounding as a solved problem. The follow-up is always how you would know it had gone wrong, and if you have not thought about that, the conversation ends there.