Your Dashboard Is Green.
The Agent Is Wrong.
AgentOps is keeping an agent correct after launch, not just online. Uptime says nothing about correctness. Here are the five signals that do.
of agentic AI projects will be canceled by the end of 2027, according to Gartner.
Gartner doesn’t say when. Our read, from the agents we operate today: plenty of them shipped fine, then drifted. Operating was nobody’s job after go-live.
Traditional Monitoring Was Built for Software That Fails Loudly
A broken API returns a 500 and pages someone. A broken agent returns a confident, well-formatted, wrong answer with a 200. Every alert you own stays quiet. Six ways that happens.
Version Drift
You pinned the snapshot, so this is handled. Except pins expire, and behavior moves under a stable version string anyway. Nothing in your repo changed. The outputs did.
Index Decay
Documents go stale, get reorganized, lose a source. Retrieval still returns results. They are just the wrong ones, and the agent answers from them anyway.
Silent Contract Changes
A renamed field doesn’t throw. The tool returns 200, the shape is new, and the model does what models do with a gap. It fills it. Nothing errored, so nothing alerted.
Borrowed Instructions
A retrieved document contains text the agent reads as instruction. In the trace it doesn’t look like an attack. It looks like the agent decided to do something reasonable.
Review Fatigue
The gate works, until it doesn’t. Reviewers approve everything, because almost all of it was fine. The control still exists on the diagram. It stopped existing in reality.
Nobody's Pager
Build team moved on. Platform team never owned it. When it drifts, the alert wasn’t missed. There was no alert, and nobody to send it to.
Most of these never produce an error. That is what makes them expensive. By the time someone notices, the agent has been wrong at scale for weeks.
Five Signals That Tell You Before Your Users Do
Uptime tells you it is running. These tell you it is still right. Instrument all five on day one. Retrofitting them means reconstructing a baseline you no longer have.
Outcome Quality
Not whether it answered. Whether it was right.
Held-out real cases, graded on a schedule. Model-graded for volume, human-calibrated for truth.
The judge is a model too. It drifts on the same schedule as everything else.
Model drift and prompt regressions, on whatever cadence you run it.
Eval noise. The same suite against the same model moves a few points run to run. Set your significance threshold before the drop, not during it.
Completion and Escalation
How often the agent finishes the job versus hands it back.
Track completion and escalation as one pair. No labeling needed, so it is the cheapest to start.
Escalation rising usually means something upstream changed shape. New format, new segment, renamed field.
Upstream change, earlier than any other signal.
Fear the good-looking direction. Escalation falling while completion rises means the agent stopped handing back the cases it should.
Cost per Completed Outcome
Not tokens. Cost per unit of work that actually finished.
Spend divided by completed outcomes, per agent, weekly.
Token spend alone hides the failure case, where cost rises because the agent retries and abandons more.
Retry storms, context bloat, and the slow erosion of a good business case.
This is the number your CFO asks for when the invoice moves. Have it before they ask.
Tool and Retrieval Health
Whether the things the agent depends on still behave as it expects.
Contract tests against every tool it calls. Retrieval quality checks against a fixed probe set.
Run them on a schedule, not only in CI. Your dependencies ship on their calendar, not yours.
Silent API contract changes and index decay.
These two are the most likely to produce confident fabrication rather than an error.
Human Override Rate
What the people at the approval gate actually do, over time.
Log every approve, edit, override, and ignore. Trend it weekly per reviewer.
Rising override means the agent has lost the room and will be switched off.
Both failure modes of human oversight, and almost nobody instruments it.
Zero override is ambiguous: a good agent, or a gate that stopped looking. Tell them apart by seeding the queue with known-bad cases and checking whether the gate catches them.
Signals 01 to 04 are engineering. Signal 05 decides whether the agent is still there in a year. The controls behind that gate are governance →
Somebody's Name Is on It
Every agent we run has a named engineer, not a queue. Right now that is Svetozar and Patrick. They know the workflow, they reviewed the evals, and the alert reaches them. Svetozar wrote the drift review that caught an index rebuild two days before anyone downstream noticed. Patrick is the reason we seed every approval queue with known-bad cases, because he does not trust a gate nobody has tested.
Operating is a staffed job. Tooling makes that person faster. It does not replace them.
On call for the agents we run. You meet them on the scoping call, and they are the ones the alert reaches.
Bring us one workflow. We prove it pays, build it properly, control it so security signs off, and keep it running after everyone else has moved on.
Running it is the clause most firms leave out of the proposal, because it is the only part that does not end.
Tools Instrument. Somebody Still Has to Operate.
LangSmith, Langfuse, Braintrust, Arize Phoenix, W&B Weave, Datadog. All good, all doing real work. We deploy them and we don’t sell a competing product.
Instrument to the OpenTelemetry GenAI conventions rather than a vendor SDK and picking wrong costs you a collector config. What no platform does is the third column.
| Layer | A platform gives you | Somebody has to own |
|---|---|---|
| Instrumentation | Yes, and better than anything you would build. Auto-instrumentation and OpenTelemetry GenAI conventions, no custom agent to maintain. | Deciding what counts as a "session" and an "outcome" in your business. The platform cannot infer it, and every metric on this page depends on it. |
| Tracing | Yes. Every step, tool call, and token, searchable. | Deciding which traces get looked at, by whom, and how often. A trace nobody opens is storage. |
| Evaluation | Yes. Harnesses, scorers, and regression suites. | Choosing the eval set, keeping it representative as the business changes, and re-calibrating the judge against humans. |
| Dashboards | Yes. Quality, cost, and latency over time. | Setting the threshold that means "act", and being accountable when it fires at 2am. |
| Alerting | Yes. Thresholds, anomaly detection, and routing into PagerDuty or an on-call rota. | Knowing what threshold on a noisy quality metric is worth waking someone for, and being the someone. Tools page a rota. They do not staff one. |
| Remediation | Partly. Versioned prompts, one-click rollback, failing traces promoted into a dataset, and the regression re-run. | Deciding the fix is safe to ship, whether the eval set that just passed still represents the business, and telling the people who depend on this what changed and why. The platform executes the rollback. It cannot own the call. |
So the question was never which platform. It is which four disciplines you staff.
Evals in CI
Gates that block a deploy, like tests do. Evals that only run manually run when someone is already worried.
A Drift Review
Thirty minutes a week, a named owner, all five signals. Beats a dashboard nobody opens.
An Incident Path
What “the agent is wrong” means as a severity, who gets paged, and a tested way to stop it.
Cost Governance
A budget per agent, an owner for it, and an alert on cost per outcome, not total spend.
Have all four staffed? Then you don’t need us for this, and we’ll say so on the call. Most mid-market teams have the tools and none of the rota.
What We Run Today, and What's Coming
We would rather publish the thinking before the price list. Here is the honest state of it.
90 days of run supportAvailable now
Included with every production agent we build. The team that shipped it operates it, five signals instrumented from day one.
Governance controlsAvailable now
Identity, scoped credentials, audit trails, human gates, and kill switches. Published in full as the Agent Governance Framework.
Standalone AgentOpsQ4 2026
Operating agents we did not build, including other vendors’ work. Tiers and response-time SLAs are in design. We are not publishing prices we might still change; when the SLAs sign off, they go on this page.
Agent in production and nobody watching these five signals? That conversation doesn’t wait for Q4.
Fair Enough. The Longer Answers.
What is AgentOps?
AgentOps is the practice of operating AI agents after they reach production: instrumenting the signals that reveal whether an agent is still correct, staffing someone to act on them, and remediating drift when it appears. It is distinct from LLM observability, which is the tooling layer that produces the traces and evals, and distinct from agent governance, which is the control layer that decides what an agent is permitted to do. Observability tells you what happened. Governance decides what is allowed. AgentOps is the operating discipline that sits on top of both, and it is the one that is usually nobody’s job.
How is this different from LLM observability tooling?
Observability platforms instrument. AgentOps operates. A platform will trace every step, run your eval suite, chart quality and cost over time, route an alert, and in most cases version and roll back a prompt. Those are real capabilities and we deploy them for clients.
What a platform cannot do is decide which traces get read and by whom, choose an eval set that stays representative as the business changes, set the threshold on a noisy quality metric that is worth waking someone for, carry the pager, or tell the business what changed and why. Every one of those is a staffing decision. The tooling question is close to solved. The staffing question is where teams actually fail.
Why can't we just use our existing APM and alerting?
Because your existing monitoring was designed for software that fails loudly, and agents fail quietly. A broken service returns a 500 and pages someone. A degraded agent returns a confident, well-formatted, wrong answer with a 200 status code, normal latency, and no error in the log.
Uptime, latency, and error rate all stay green through model drift, index decay, a renamed field in a downstream API, and a review gate that has quietly become a rubber stamp. Your APM is not broken and you should keep it. It simply cannot see the class of failure that matters most here, because to your APM nothing failed.
How do you evaluate an agent that has no obvious right answer?
You define a rubric rather than an answer key, then hold two things steady: a held-out set of real cases that stays representative, and a grader whose behaviour you check. Model-graded evaluation carries the volume, but the judge is a model too and drifts on the same schedule as everything else, so it gets recalibrated against human labels periodically.
The part most teams skip is variance. The same suite against the same model moves a few points run to run. If you have not decided what counts as a significant drop before you see one, you will spend the incident arguing about whether it is real rather than fixing it.
What does it cost to run an agent properly?
Two costs, and the second one surprises people. The first is inference, which you should measure as cost per completed outcome rather than tokens. Token spend alone hides the failure case, where cost rises precisely because the agent is retrying and abandoning more.
The second is the human. Somebody has to run the drift review, own the threshold, take the alert, and do the remediation. That is a real allocation of a real engineer, and pretending a dashboard replaces it is how agents end up unowned. Our standalone AgentOps tiers are in design for Q4 2026 and we are not publishing prices until the SLAs are signed off.
Can you operate an agent your team didn't build?
That is what the standalone service is for, and it launches Q4 2026. In the meantime we do it inside builds and rescue engagements, so if you have something in production right now that nobody is watching, the conversation does not need to wait.
The honest constraint: we need to be able to instrument it. If the agent is a closed vendor product with no trace export and no eval hook, there is a limit to what anyone outside that vendor can operate, and we will tell you that before you spend anything.
Bring Us One Agent You Can't See Into
Thirty minutes with a senior engineer. We’ll walk the five signals against the agent you actually have, and tell you which ones you could instrument yourself this month, for free.
What it does, what it runs on, and who looks at it today. Three answers, that is the whole prep.