Checking work that always sounds right

The trap

If checking the agent costs as much as doing the task, you have gained nothing. So the goal is not to re-do the work — it is to design tasks whose output is cheap to verify and expensive to produce. That asymmetry is where the value actually is.

Ask for output you can check mechanically

Where possible, request a form that a computer can validate: a list that must have the same number of rows as the input, a set of file paths that must all exist, structured data that must parse. Then check it mechanically, every time. Machine-checkable output is worth accepting a slightly worse format for.

Ask it to cite its own source

“For each claim, quote the sentence from the material you took it from.” This works better than it should, for a mechanical reason: an invented claim usually cannot produce a real supporting quote, so invention becomes visible without you knowing the subject. Then spot-check the quotes against the source — and check that the quote actually says what the claim says, which is where the interesting failures are.

Salt it with a known answer

Include, in the batch, one or two items whose correct answer you already know. If it gets those wrong, discard the batch. This is cheap, it scales, and it is the only technique here that keeps working when you no longer have time to read everything.

Check the edges, not the middle

Given fifty results, do not read the first five. Read the shortest, the longest, the empty one, the one with the odd character, and the one that took strangely long. Failures cluster at the extremes; the middle is where it does well.

Watch for silent scope drift

Ask for twelve summaries, get eleven and a note that one email was “not relevant”. That judgement was never yours to delegate. Always verify counts. Agents drop items quietly far more often than they produce visibly wrong ones, and a count is the cheapest check in existence.

Keep the raw material

Whatever the agent produced, keep the input alongside it. When you find an error in three weeks, you need to know what it was working from — and if you ever change the brief, you need the old material to test the new brief against. Store the input, the brief, and the output together.

One thing not to do

Do not have a second agent check the first one and treat that as verification. It catches some things and it is genuinely useful as a first filter, but two models sharing similar training share similar blind spots, and a confident second opinion feels like corroboration when it is not. Use it to triage; do not use it as the final check on anything that matters.