polycratia

In production your agent didn't fail. It succeeded against a smaller world than it thought it had.

What usually opens the gap is authorization scope, not reasoning quality.

In testing, the agent runs under one credential, almost always the broadest one in the system. Every read returns the whole table, and the plan gets built on a complete picture.

In production it runs on behalf of many principals, and each one sees a different slice. A role-scoped query does not error when the caller lacks access to half the rows: it returns the half they can see, with a 200.

That is what breaks agents specifically. A truncated list is a syntactically valid answer. The agent cannot tell "there are three open items" from "you may see three of the eleven open items", so it reasons confidently on a partial world. Then it closes the wrong thing and reports completion, escalating nothing.

Deterministic code has the same blind spot. The difference: a human reads the resulting screen and senses something is missing. An agent reads it and plans.

How I handle it: scope belongs in the tool contract, not in the infrastructure underneath it. Every read the agent makes returns what was filtered and why. Anything with a non-empty filter reason cannot be the basis for a terminal action, so it routes to a human instead. Permission-shaped emptiness and real emptiness get different types.

It is the same discipline I apply to compliance moderation tooling: the reviewer has to know whether a field is clean or merely invisible to them. Agents need that distinction more, not less, because they do not pause to wonder.

So before you blame the model, run your evals under a narrow principal instead of an admin token. The failures probably show up on the first pass.

I write more about this kind of production detail on my blog: https://polycratia.com/c/01c2c505

If you're running agents against multi-tenant data, go look at whether your tool layer tells the agent what it wasn't allowed to see...

react

$ new-project --brief

or email hey@polycratia.com