From Plausible to Proven: Working With LLMs Without the Magic

Imagine you ask a friend for directions to Elisabeth’s place for dinner.

Without missing a beat, they give you a clear route. Turn left at the bakery, second right, the blue house at the end. Confident. Detailed. Easy to follow.

You drive there. You knock. A stranger answers.

Nothing was wrong with the directions, exactly. They were a perfectly good route — to the house Elisabeth lived in last year. Your friend never heard she moved. They filled the gap with the address they remembered, and delivered it with the same certainty they’d have used for a fact.

One thing they didn’t know changed the whole outcome. And you couldn’t tell from how sure they sounded.

That is the failure at the centre of working with a large language model. It ties the last two articles together better than another diagram could. What an LLM is actually doing showed the engine: an LLM predicts plausible next words, not verified facts. How LLMs remember things showed the desk: the model can only use the instructions, files, messages, and tool results that actually reach it.

Put those together and the practical consequence is hard to avoid.

An LLM is not an oracle. It is a fast, confident helper that will often answer fluently even when the key fact is missing, using whatever it happens to have in front of it. Its training supplies broad, general experience, like a friend who knows the town well. What you put in front of it right now is the errand and the details it can see. Tools let it go look things up or take action in the real world. None of that, on its own, tells you whether the answer is actually right.

That takes checking against something outside the model.

The answer looked right. What was missing?

The stranger at the wrong door is not a rare glitch. It is the normal shape of the mistake.

When I was drafting that first article, the model gave me several explanations that read beautifully and were subtly off. A cause stated too confidently here, a detail smoothed over there. Not gibberish. Nothing that waved a red flag. Just claims delivered in the same steady voice as the correct ones, a couple of them quietly wrong.

That is the problem in miniature. The wrong answers do not look wrong. They look exactly like the right ones.

The model’s core job is to continue text plausibly. A polished paragraph and a fabricated citation can come out of the same machinery. OpenAI’s research on why language models hallucinate traces part of this back to next-word prediction and part to an evaluation culture that often rewards guessing more than abstaining. If a confident guess has a chance of scoring and “I don’t know” does not, guessing is what gets reinforced.

So how sure it sounds tells you nothing about where the answer came from. The answer may well be right. The question is what makes it right.

For factual work, I now want to know four things:

• Which source supports the claim?
• Is that source current and primary?
• Does it say what the answer claims it says?
• Which parts are still inference?

That is more useful than asking the model how confident it is. A stated confidence can track accuracy loosely, but models tend to run overconfident, and either way it is not provenance. It’s the friend telling you again, in a firmer voice, that it’s the blue house. A source you can open is a different thing entirely. It’s the address on the contact card.

The same split shows up everywhere, not just in facts. A tidy-looking answer is the route. A check that passes tells you the route is drivable. Actually arriving, the thing you wanted, is a separate claim you still have to confirm.

Plausible is a start.

It is not done.

A prompt is only the work order

Prompt advice tends to focus on wording. Give the model a role. Add a magic phrase. Tell it to think harder. Rearrange the same request until the response feels better.

Wording matters, but no amount of it can reveal a fact the model was never given.

Asking your friend for directions in a more polite, more detailed, more insistent way would not have helped. They would just give you the wrong house more eloquently. What was missing was not better phrasing. It was Elisabeth’s new address, and ideally her text saying she’d moved.

A useful request contains five things:

1. The outcome: what should be true when the job is finished?
2. The constraints: which rules must the answer respect?
3. The source of truth: where should it look instead of guessing?
4. The action boundary: what is it allowed to check, change, or do?
5. The acceptance test: what will count as actually done?

For the dinner errand, that request might be:

Get me to Elisabeth's place for dinner tonight, arriving before 7.
Check her current contact card and our recent messages first —
she may have moved. Give me the route from home.
I'll know it worked when I'm standing at the right door.

Hand that to a capable helper with the contact card and messages in reach, and the answer comes back grounded instead of guessed:

Checked her contact card: 8 Rowan Lane (updated last month).
Her message says come to the back gate.
Route from home: 19 minutes, arrives 6:41pm.
You'll confirm it worked when someone you know answers.

No magic words. Just the information needed to do the job.

Anthropic calls this wider problem context engineering: choosing the smallest set of high-signal information that makes the right behaviour likely. It’s far more than the question you type. The standing instructions, the available tools, the files, the earlier conversation, and every tool result all compete for the same limited attention.

A bigger pile is not a better brief. Handing your friend a phone book and saying “she’s in here somewhere” is not the same as handing them the one address. The useful page is in there, buried under everything else.

How to actually ask

The dinner brief is a template you can reuse. Strip out the story and it’s five labelled lines: outcome, constraints, source of truth, action boundary, acceptance test. Here it is on a real task instead of an errand:

# Outcome
Find why `parseInvoice()` returns null for some PDFs and fix it.

# Constraints
Don't change the function signature. Keep the existing tests passing.

# Source of truth
Read src/parse.ts and the three failing fixtures in test/pdfs/.
Don't guess at the PDF structure — open them.

# Action boundary
Propose the fix as a diff first. Don't touch anything outside src/parse.ts.

# Acceptance test
All tests green, and the three fixtures that returned null now parse.

That is not a magic phrase. It’s the same brief, aimed at a different job. This is what Anthropic means by treating the model like a brilliant new employee who lacks context on your norms: spell out the job, the boundaries, and what “done” looks like, because it can’t infer any of them from the town it grew up in.

Three moves make almost any brief land better.

Show it, don’t describe it. If you want a particular shape of answer, the fastest way to get it is to hand over a couple of finished examples. Describing the format in prose is guessing; a worked example is the address. Both Anthropic and Google push this hard. Anthropic suggests three to five examples, and it’s the oldest trick in the book.

Turn each raw log line into a one-line incident note. Match this shape:

2026-04-02 03:11 ERR db timeout on /checkout
→ Sev2 · Checkout · DB timeout · 03:11 · needs on-call

2026-04-02 03:19 WARN retry succeeded /checkout
→ Sev3 · Checkout · Recovered after retry · 03:19 · monitor

Now do these:
[paste log lines]

Pin the output format. If you’re going to read the answer at a glance, or feed it to something else, say exactly what shape it comes back in. Ask for the four fields and nothing else. Ask for a diff, not a description of a diff. Vague requests get prose; specific ones get the thing you can use.

Ground it in a source. When the answer has to be right, don’t let the model reach into memory. Give it the text and pin it to that text:

Answer using only the passage between the <ref> tags. If the answer isn't
there, say "not in the source." Quote the exact sentence you relied on.

<ref>
[paste the actual policy / doc / thread here]
</ref>

Question: what's the refund window for damaged goods?

OpenAI calls this providing reference text; it’s the single biggest thing you can do to stop a confident guess. The “quote the sentence” line matters — it forces the answer to point at something you can check, instead of sounding sure.

None of this is a trick. It’s the difference between telling your friend “she lives around there somewhere” and handing over the address, the note about the back gate, and the time you need to arrive.

When the same brief keeps coming back

Some briefs you only write once. Most of the good ones you end up writing again.

You have dinner at Elisabeth’s most Fridays. By the third visit you’re tired of repeating yourself: check the current card, use the back gate, she likes you there by seven. That’s not a fresh errand each time. It’s a standing note you’d rather write once and hand over whenever it’s relevant.

That standing note is a skill. Claude added them in late 2025, and by mid-2026 they’d stopped being an Anthropic-only feature — ChatGPT and Codex adopted the same format, so a skill written for one can be dropped into another. A skill is a folder with a SKILL.md file inside: a name, a one-line description of when it applies, and the instructions for doing the job. Optionally a few examples, reference files, or a script the model can run. Anthropic’s own framing is the plainest one — building a skill is “like putting together an onboarding guide for a new hire”. Everything the five-line brief spells out for one task, a skill spells out once for a task you keep repeating.

The clever part is when it loads. The model doesn’t read every skill on every message — that would bury the desk under standing notes you don’t need right now. It reads only the name and description of each, and pulls in the full instructions when the description matches what you’re actually doing. Anthropic calls this progressive disclosure: the one-line summary is on the desk, the rest stays in the drawer until the job calls for it. Ask about a spreadsheet and the spreadsheet note comes out. Ask about dinner and it doesn’t.

So the description is the load-bearing line. “Formatting” is too vague to fire at the right moment; “Convert raw log lines into one-line incident notes, Sev-tagged, for the on-call channel” tells the model exactly when to reach for it. Same discipline as pinning the output format — you’re writing the label on the drawer, not just what’s inside.

A skill doesn’t buy you anything a good brief didn’t. It’s the same evidence, the same boundaries, the same definition of done — written down once instead of retyped every Friday. And it earns its place on the same test as everything else here: only when the errand actually repeats. A one-off doesn’t need a standing note. A phone book of skills you never open is just the desk buried again.

Let it inspect before it invents

Pasting the text in works when you already have it. Often you don’t. The document might be too big, or change by the hour, or you simply don’t know which page holds the answer. Then you stop pasting and let the model go and look.

For the dinner errand, that means checking the contact card before setting off, not trusting a year-old memory. For a question about how some service works today, it means opening the current documentation instead of recalling an old version. For “is the shop still open,” it means calling the shop, not guessing from what was true last winter.

This is where tools earn their keep. They let the model go and pull real evidence at the moment it needs it. That evidence lands in front of the model, and it carries on with better material than it started with: the same helper, now looking at the right address instead of remembering the wrong one.

Going straight to the source is often simpler than building elaborate machinery to search for something whose location you already know. If the answer is in one document, open that document. If it is one entry in a list, look up that entry. If it is today’s price, ask whoever owns the price. Full-blown search systems earn their place when you genuinely have a large pile to comb through. When the right drawer already has a label on it, they are just plumbing you didn’t need.

The rule travels outside software too:

• Give a summarisation task the complete document, not a description of it.
• Give research work current sources and require links back to them.
• Give scheduling or pricing questions live tools, not permission to guess from old weights.
• Give writing work the actual audience, constraints, and examples that define the voice.

Inspection does not guarantee a correct result. It removes a whole class of avoidable invention.

Click through the dinner errand with each layer added in turn:

Head to 14 Maple Street.
Turn-by-turn route ready. 12 minutes.
"Confident this is the one."

The first answer is plausible because a confident route to a remembered address is normal. The second is grounded because the helper can now see the current contact card and her note about moving. The third has receipts: the route is drivable and the timing works. Only the fourth walks up and knocks, checking the thing that actually mattered.

Same helper. Better work loop.

When should it be allowed to act?

Tools do not only look things up. They can change files, send an email, buy something, delete data, or push a change live to real users.

That changes the stakes. A wrong paragraph is annoying. A wrong action out in the real world has blast radius.

I find it useful to separate four levels of access:

1. Read the sources it has been authorised to read for this task.
2. Propose a plan, or draft something for you to review before anything happens.
3. Do reversible work on its own and check its own results.
4. Stop and ask before anything sensitive or consequential: touching data it wasn’t cleared for, spending money, or an action that reaches other people or can’t be undone.

The boundary moves with the job. Fixing a typo in a private draft is cheap to undo and easy to eyeball. Publishing an article changes what readers see. Sending a message reaches another person. Deploying or deleting touches a shared system. Paying moves money.

OpenAI’s practical guide to building agents recommends human intervention for high-risk actions and after repeated failures. That is not a patch for a stupid model. It is ordinary operational control. We put approvals around consequential actions when humans operate systems too.

The useful question is not “How autonomous should AI be?” in the abstract. Ask three smaller ones:

• How bad is a mistake?
• How easy is the action to reverse?
• How clearly can we verify the result?

Cheap, reversible, visible work can run with a loose leash. Expensive, irreversible, or hard-to-observe work needs a gate.

The model can ask for the key.

The application decides whether to hand it over.

A check has to see the failure

Checking that the route was valid answered a real question: is there a drivable road that gets me to 8 Rowan Lane by seven?

Yes.

It did not answer a different question: is anyone I’m looking for actually behind that door?

No amount of route-checking could settle that. The map check was not bad. It was aimed at the wrong failure: whether the road works, not whether anyone I wanted was there.

This is one of the most practical changes the first two articles made to how I think about AI work. Do not just ask for verification. Choose a check that can actually see the failure you’re worried about.

If the answer makes a factual claim, open the source it cites and read whether it says that. If it does a calculation, redo a few cases by hand. If it changes something mechanical, exercise the thing that changed and watch it behave. If it drafts a message that will be sent, read the real preview. If it changes something live, go look at the live thing afterwards.

A tool’s output helps, but it is still just a receipt handed back to you. The map can be out of date. A quick test can skip the exact case you care about. A search can return a page that’s nearby but doesn’t actually say what you needed. A system can cheerfully confirm it received your request a moment before the real work falls over downstream.

Evidence has a scope.

“The route is valid” is evidence that the route is valid. Nothing more.

So what is the human still for?

The model can do a lot of the mechanical middle now. It can search, compare, draft, edit, run commands, interpret errors, and try again faster than I can do each step by hand.

That does not leave the human with nothing. It moves the job.

The human defines the outcome. The human decides which source deserves authority. The human names constraints the model cannot discover. The human sets the action boundary, judges whether the evidence is strong enough, and owns the final sign-off.

This is not a grand claim about which species can truly reason. It is a responsibility boundary for practical work.

A more capable model can make the middle faster and recover from more mistakes. It cannot make an outdated document current. It cannot turn a shallow test into a useful one. It cannot decide that a business risk is acceptable unless someone gave it the policy and the authority to apply it.

The work around the model is not scaffolding we will eventually throw away. It is where the source of truth, permissions, and definition of done live.

That is the system.

The operating loop

Here is the version I use:

1. Brief: state the outcome, constraints, and definition of done.
2. Inspect: let the model read the relevant environment before it proposes.
3. Ground: fetch current, authoritative evidence for claims that matter.
4. Act: allow only the actions appropriate to the risk and reversibility.
5. Verify: use a check that can see the failure you care about.
6. Sign off: keep a human decision where the consequence deserves one.

The loop can be tiny. A quick, low-stakes question may need only a clear brief and one glance to confirm. A high-stakes change may need the current documents, the right tools, a real test, a look at the live result, and someone’s explicit sign-off before it goes out.

Same machinery. Different consequences.

The directions were useful. They told me there was a good road to somewhere. They did not tell me it was the somewhere I wanted.

The route was valid.

The door was still the wrong one.

Reliable work starts when you stop trusting how sure the answer sounds and walk up and knock.

No magic. Just a clear request, the right information in front of the model, and a check that can actually come back wrong.

The interactive example uses hand-authored text and simulated results. It runs no model, map service, or network request. The scenario is illustrative. The failure sequence, a confident answer built on a fact that was never supplied, is the real one this whole series is about.