For about a year we tried to build Novrex AI as a single, very capable model with a giant tool catalog. It worked beautifully in demos. It started falling apart in production around the time we crossed a few thousand workflows a day.
The failures were not random. They had a pattern: the more objectives we asked one agent to balance — answer the customer, update the CRM, check inventory, draft a quote, log to analytics — the more often it would skip steps, hallucinate field names, or quietly choose the wrong tool.
A single agent is a generalist with a deadline. That is a recipe for shortcuts.
Specialisation as reliability
We rebuilt Novrex AI as a small team of specialised agents, each with a narrow domain and a tight tool surface. There is a lead-routing agent, a pricing agent, an inventory agent, a reconciliation agent, an analytics agent. None of them know about each other. They only know how to do their one thing exceptionally well.
Above them sits an orchestrator. The orchestrator is the only component that sees the full request. It plans the steps, dispatches them to specialists, collects results, handles fallbacks, and writes the final response.
The orchestration patterns that matter
Three patterns do most of the work in our orchestrator.
1. Plan first, act second
Before any tool call, the orchestrator produces a structured plan: ordered steps, expected outputs, abort conditions. The plan is cheap to generate, easy to validate, and trivially loggable. When something goes wrong in production, you debug the plan, not the conversation.
2. Typed tool boundaries
Every tool exposed to an agent has a strict input/output schema enforced at the runtime layer, not in the prompt. We treat the LLM as untrusted input — same way you would treat a form post. If a model returns a malformed payload, the runtime rejects it and asks the model to retry with a structured error. We do not "just hope" it formatted JSON correctly.
3. Idempotency by default
Every action an agent takes carries an idempotency key. If the orchestrator retries — and it will — the underlying system de-duplicates. This single discipline eliminated the vast majority of "agent created two invoices" incidents we used to see.
What we monitor
Latency and cost are the obvious metrics. The ones that actually predict quality are different:
- Plan stability — how often does the same input produce the same plan? Drift here is your first early warning.
- Tool selection accuracy — measured against a curated golden set, not vibes.
- Refusal rate — when an agent declines to act, is it doing so for the right reasons?
- Human override rate — the percentage of agent decisions a human reverses. This is the closest thing we have to a quality north-star.
What still keeps us up at night
Two things, honestly. The first is silent regressions — model providers ship updates that subtly change behaviour, and you only find out when a downstream metric moves a week later. We mitigate with golden-set evals on every routing decision, but the failure mode is still real.
The second is operator trust. Agents that work 99% of the time are still agents that fail 1% of the time, and the failures look very different from the deterministic bugs operators are used to. We have invested heavily in explainability — every agent decision exposes its reasoning trace and the documents it consulted — because trust is built one transparent decision at a time.
If you are building production agentic systems, prioritise observability and idempotency before you optimise prompts. The prompts will keep changing. The discipline is what compounds.