Fluent is not the same as correct

The core property

A language model produces text that plausibly follows from the text before it. It is extraordinarily good at this. It is not consulting a database of facts and reporting what it found, even when the output looks exactly like it did.

Almost every surprising agent behaviour follows from that one sentence.

What it explains

Confident invention. Ask for a file path, a citation, a command-line flag, or a person's title and you may get a beautifully formed one that does not exist. The model is not lying — there is no internal notion of “I do not know” to report. A plausible-looking answer is exactly what it was built to produce.

Confidence is uninformative. Human confidence correlates loosely with human accuracy, so we read confident writing as a signal. From a model, the tone is generated the same way whether the content is right or wrong. You must stop reading fluency as evidence. This is a hard habit to break and it is the single most valuable thing in this course.

It agrees too readily. Push back on a correct answer and it will often fold and produce a wrong one, because agreement is a plausible continuation of disagreement. Test this once on something you know cold — it recalibrates you permanently.

Filling gaps in ambiguity. Give it an underspecified task and it will not stop and ask. It will pick an interpretation and proceed, and the interpretation will be reasonable, and it will not be flagged as a choice. This is why the ambiguity rule from Robot Readiness matters even more here.

What it is genuinely good at

Reshaping text you supply. Summarising material that is in front of it. Classifying things into categories you defined. Drafting a first version. Explaining an error message. Finding the odd one out in a list.

Notice the pattern: it is strongest when the material comes from you and the task is transformation rather than recall. That is the shape to aim your work at.

What it is unreliable at

Recalling specific facts, especially recent, local, numeric, or rare ones. Arithmetic over more than a few steps. Anything requiring it to know what is true right now. Knowing the boundary of its own knowledge — it will not tell you when it has left it.