Platform · Module
AI Observability
See what every AI call and agent run did, what it cost and where it failed
AI Observability traces every model call, agent run and Copilot turn with cost, quality scores and guardrail events. It feeds evaluations, FinOps, governance and the ROI Ledger.
Every call traced
Each model and tool call becomes an OpenTelemetry span with trace id, tenant, actor, goal, workflow and Action ids, model, provider, prompt version, tokens, cache hit, latency, cost and guardrail events. A tenant-scoped trace viewer links each trace to the Action log, so you can go from "what did the agent do" to "what did the model see" in one click. A daily cost report runs per tenant and per agent from Wave 0.
Quality, not just uptime
From Wave 1, scores are recorded for evaluations, user feedback, groundedness and schema validity. A guardrail event log captures injection detected, PII masked and output rejected events.
Agent run timelines
An agent run timeline shows the plan, steps, tool calls, approvals and outcome, including the connector health checks the agent made before acting.
Finding problems early
Planned for Wave 2:
- Failure clustering groups failures weekly by error signature and intent, with example traces and links to evaluation cases.
- Drift alerts flag score regressions, cost-per-task spikes and latency regressions per route.
- AI incidents can be created from a trace, with severity and a link to kill switches.
- A tenant-facing dashboard shows usage, cost, quality and incidents.
- Raw traces are retained for 30–90 days (estimate), with aggregates kept longer.
The basic level is included. Deeper analytics live in AI FinOps and AI Governance in the AI Workforce app. Forensic replay of an agent run and online A/B experiments on prompts and routes are planned for Wave 3.
Features
What AI Observability includes
OpenTelemetry spans
Trace, tenant, actor, Action, model, prompt version, tokens, latency, cost and guardrail events per call. W0.
Trace viewer
Tenant-scoped, linked to the Action log. W0.
Daily cost report
Per tenant and per agent. W0.
Scores
Evaluation scores, user feedback, groundedness and schema validity. W1.
Guardrail event log
Injection detected, PII masked, output rejected. W1.
Agent run timeline
Plan, steps, tool calls, approvals, outcome and health checks. W1–W2.
Failure clustering
Weekly clusters by error signature and intent. W2.
Drift alerts
Score, cost and latency regressions per route. W2.
AI incidents
Created from a trace, with severity and kill-switch link. W2.
Tenant dashboard
Usage, cost, quality and incidents. W2.
Forensic replay and A/B tests
Replay an agent run; split traffic between prompt or route versions. W3.
Workflow
Investigate a bad agent reply
- 1
A user flags an agent reply.
- 2
Admin opens the trace from the Action log.
- 3
The timeline shows retrieved sources and guardrail events.
- 4
Admin creates an AI incident and, if needed, triggers a skill kill switch.
AI
AI skills and Digital Employees
Failure clustering
Weekly clusters with example traces and eval-case links. W2.
Drift alerts
Signals score, cost and latency drift; optional auto-demotion through AI Governance policy. W2.
Groundedness and schema scoring
Scores outputs automatically. W1.
Digital Employees that use this module
- AI Agent Governance Auditor
- AI SRE On-Call Assistant
They act through the same governed Actions as people; risky steps wait for approval.
Frequently asked questions
Can I see what an agent did step by step?
Yes. The agent run timeline shows plan, steps, tool calls, approvals and outcome, linked to the Action log.
Is AI Observability an extra cost?
The basic level is included. Advanced analytics come with AI FinOps and AI Governance in the AI Workforce app.
How long are traces kept?
Raw traces are planned for 30–90 days (an estimate), with aggregates kept longer.
See AI Observability in your workflow
Tell us about your team and the outcomes you want. We will walk you through the apps, the Digital Employees and the plan for your industry.