They ask: How were large language models actually trained? Somebody had to label all that text, surely.
You say: No, and that is the whole trick. Labelling the internet by hand is impossible. It would take lifetimes and cost more than anyone has.
So they made the data label itself. Take a sentence, hide one word, make the model guess it, then uncover the word and let the model see whether it was right. No human in the loop. You can run that on everything ever written.
Do that enough times and the thing gets extremely good at guessing what comes next. That is what a language model is. A machine that has played fill in the blank more times than you can count.
And it explains the behaviour you see later. The model is not consulting anything when it answers you. It is still finishing the sentence, the way it was trained to.
The answer that ends the conversation early: "it read the internet." Everyone says that. It did not read anything. It played a guessing game against text, billions of times, and the guessing is still what it does when it talks to you.
They ask: Why do these models produce wrong code?
You say: Because of where the code came from. These systems learned to write software by consuming enormous amounts of public code, and public code is not vetted. Anybody can publish. Plenty of it carries bugs, bad structure, and patterns that were wrong when they were written.
I think of public repositories as social media for code. Some of it is excellent. Some of it is somebody's abandoned weekend project. The model learned from all of it without knowing which was which.
So when it produces something confident and wrong, it is not malfunctioning. It learned that from us.
This is also why the wrong code often looks so plausible. It is not random. It is a pattern that appeared often enough in public work to be worth imitating, including the patterns that were mistakes.
The answer that ends the conversation early: "the model hallucinates sometimes." True and shallow. Connect the output back to the training data and you have shown you understand the mechanism instead of the symptom.
They ask: What is the fundamental limitation of a large language model?
You say: It predicts. It does not retrieve.
A database looks the value up. Ask it for a record that is not there and it returns nothing. That is the correct behaviour and you can build on it.
A language model does something else. It produces the most probable next piece of text given everything before it. When the answer sits in the training data thousands of times, the prediction and the truth land in the same place and it looks like the machine knows something. When the answer is thin or absent, it still predicts, because predicting is the only thing it does. It has no mechanism for returning nothing.
A database gives you the number. A model tells you what the number probably looks like.
Everything else in this book follows from that sentence. The controls you put around an agent, the evidence you demand, the reason you cannot accept a summary. All of it is a response to a machine that has no way of returning nothing. If you take one sentence out of this book, take that one. It is the sentence the rest of the interview will be built on.
The answer that ends the conversation early: calling it a search engine, or saying it "looks things up." Everything else you say afterwards is built on a wrong foundation, and a good interviewer will hear it immediately.