They ask: What are evals?
You say: It is the word the AI side of the industry uses for evaluating whether a model or an agent is producing acceptable output. In practice it is quality assurance applied to something that does not behave the same way twice.
I say that plainly in interviews because it is true and because it repositions the whole conversation. Teams describe evals as a new discipline they need to build. It is testing, done against a system that gives you a different answer to the same input, so you measure a rate rather than assert a single result.
The people best placed to do it are the ones who already think in terms of coverage, edge cases and evidence.
The answer that ends the conversation early: treating it as an unrelated speciality you would need to be trained in. You are describing a gap between yourself and the job that mostly is not there.
They ask: How do you verify what an agent produced?
You say: I require the artifact. The execution output, the screenshot, the report file. Something that exists independently of the agent's description of it.
The reason is specific. When an agent says the tests passed, that sentence was produced by the same process that produces everything else it says. It is a prediction of what a report would sound like. It is not a reading of a result.
I learned that on a morning when an agent told me all tests passed and had in fact deleted the work. Nothing in the transcript looked wrong. The folder was empty.
So the rule I work by is that if I cannot see the thing that would have failed, I have not verified anything.
The answer that ends the conversation early: "we review the agent's output." Review which output. Say artifact, say log, say screenshot, and the room knows you have been burned and learned something.
They ask: What changes about traceability when agents are writing the code?
You say: It stops being paperwork and becomes the control.
A traceability matrix maps every requirement and every acceptance criterion to the test that proves it. Historically people treated it as an audit artifact, something you produced because a process demanded it.
With agents in the loop, two new things happen. Requirements get generated and edited by AI, so they drift. And agents add behaviour nobody requested, because it seemed reasonable in the moment. Now you have functionality with no requirement behind it and requirements with no test in front of them.
If every criterion has a test attached, an unrequested feature stands out immediately. It is the thing with no line pointing at it.
The answer that ends the conversation early: calling it documentation overhead. That was defensible when humans wrote every line deliberately. It is not defensible when the code volume went up and the requirements became editable by a machine.
They ask: What do you think of self-healing tests?
You say: Useful, and the risk sits in the same place as the benefit.
Self-healing means the agent notices a test failed because an element moved or was renamed rather than because behaviour broke, and it updates the script and carries on. That saves genuine time. Interface changes break enormous numbers of tests without anything being wrong.
The problem is that from the outside, a healed test and a suppressed defect look the same. Both end with a green run. If the agent decides a change was cosmetic and it was not, it has repaired the test around a real bug and reported success.
So I want healing to be visible. What changed, what it was before, why it was judged cosmetic. Healing that happens silently is a system that hides exactly the failures you built it to find.
The answer that ends the conversation early: "it makes the suite more stable." Stability is not the goal. A suite that never fails is not stable, it is broken in a way that is comfortable.