AISTORSServicesManaged AI and cloud operations
Service 09The same four measures, every month.
For teams with an AI system already live and nobody whose actual job is keeping it right. Most providers stop at handover. Models get retired, accuracy drifts, and the person who understood it moves team. We run it, report against the baseline recorded at build time, and give you a decision each month rather than another dashboard.
Book a 30-minute call
If the left column is not you, say so on the call and we will tell you what would help instead.
Four things go wrong after go-live. Each has a named owner and a written threshold before we take the system on.
What we do. Tracked from the announcement rather than from the outage. The successor is scored against your fixed evaluation set, behavioural differences are reported, thresholds re-established, and cutover scheduled with a rollback path.
What we do. Drift detection against the baseline recorded at build time, with thresholds agreed in writing and an escalation path that names a person rather than a shared mailbox.
What we do. Monitoring is the input, not the product. The output is a decision every month and a change every quarter. Unchanged reports on a live AI system mean nobody is looking hard enough.
What we do. Usually we still can, after an onboarding assessment that establishes the measures nobody recorded, finds the gaps in logging and access control, and says plainly whether it can be operated or needs remediation first.
Reported against the baseline recorded in the Diagnostic, so the comparison stays honest.
A measure below threshold is a reason to pause the system, and the report says so on the same line.
Task completion rate, human intervention rate, cost per completed task and accuracy against benchmark. Each gets a baseline value and a threshold, and the threshold is a contract term rather than a dashboard setting we can quietly widen later.
Logging of inputs, outputs and decisions; cost attribution per feature; and a fixed, versioned set of real cases labelled by your team. That set is the only thing that makes a model change measurable rather than a matter of opinion.
Drift, accuracy, intervention rate and cost run continuously with alerting. The report is monthly and always the same four measures in the same format, so twelve months of it is a comparable record rather than twelve different documents.
Deprecation notices are tracked as they are published. When a version is retired, the successor is scored against your evaluation set, differences are reported, thresholds are re-established and cutover is scheduled with a rollback path.
Where the system is over-escalating, where a cheaper model would score the same, where a threshold is set too loose, and what should be retired. This is the session that changes things rather than reporting them.
Built on your platform's own observability, so nothing of ours has to keep running for the reports to work.
Azure
Azure Monitor and Application Insights, Log Analytics, Azure AI Foundry evaluations and content safety, Entra ID access reviews, Azure Policy.
AWS
CloudWatch and X-Ray, Bedrock model evaluation and invocation logging, Bedrock Guardrails, IAM Access Analyzer, CloudTrail, Config.
Google Cloud
Cloud Monitoring and Logging, Vertex AI Model Monitoring and evaluation, IAM Recommender, Cloud Audit Logs.
DigitalOcean
Monitoring and alert policies, managed database metrics, plus OpenTelemetry and Grafana where the native surface is thinner than the workload needs.
Where an open-source stack is the better answer, we build on OpenTelemetry and Grafana rather than on anything proprietary to us. Flexera's 2026 research notes that managed service providers remain critical for handling complexity while customers retain ownership of governance and cost accountability. That split is the correct one and it is how this engagement is structured.
You do not need these numbers to recognise the problem. They are here because somebody in your approval chain will ask.
150,000
agents or more will be in use at an average global Fortune 500 enterprise by 2028, up from fewer than 15 in 2025. Gartner published six steps for managing the resulting sprawl.
Source: Gartner, 28 April 2026.
28%
of organisations have an incident-response playbook for when an AI system fails. 23 percent conduct adversarial testing.
Source: UST, Enterprise AI at Scale, 2026; global survey of 510 senior leaders.
40%
of organisations deploying AI are forecast to use AI observability to monitor model performance by 2028. Which means most still will not, two years from now.
Source: Gartner, 12 May 2026.
The incidents are already happening. The OneTrust 2026 AI-Ready Governance Report found that 86 percent of organisations had experienced AI-related incidents, that 87 percent encourage AI agent use while only 47 percent have clear governance, oversight and controls in place, and that 28 percent had two or more incidents in the past year in which AI systems or agents took unapproved actions.
Four things move on their own. Each has a detection method, a cadence and a defined response.
| What changes | How it is detected | Cadence | What happens when it trips |
|---|---|---|---|
| Model version | Provider deprecation notices tracked against a pinned version register, plus a fixed evaluation set re-run on every provider change. | On announcement, and monthly | Successor model scored against the same evaluation set before cutover. Thresholds re-established rather than assumed. Migration is planned work, not an outage. |
| Data distribution | Input drift monitoring against the distribution recorded at build, plus accuracy scored on a versioned set of real cases labelled by your team. | Continuous, reported monthly | Below threshold is a reason to pause the system, and the monthly report says so in the same line rather than in a footnote. |
| Cost per task | Token, request and GPU cost attributed per feature and per team, indexed to the cost per completed task recorded at baseline. | Monthly, with anomaly alerts | Cost regression is treated as a defect. Gartner forecasts inference cost per agentic workflow rising more than fivefold through 2028, so this is a trend, not a spike. |
| Permissions and access | Credential expiry register, scope review against the actions the system actually performs, and audit logs of every write. | Quarterly, and on any scope change | Unused permissions are withdrawn. Extension is a decision someone makes and signs, not a default that survives because nobody looked. |
Usually yes, and this is a common way engagements start. It requires an onboarding assessment first, because we cannot report against a baseline that was never recorded. That assessment establishes the current measures, finds the gaps in logging and access control, and tells you honestly whether the system is in a state that can be operated or needs remediation first.
Stated in writing per engagement, with coverage hours, response commitments and escalation paths named. We deliberately do not offer round-the-clock cover from a senior founder-led practice, because that would be a promise the structure cannot keep. Where you need it, it is scoped with the associate network and the individuals are disclosed to you by name before they touch your systems.
The boundary between what the system may decide alone and what always requires a person is agreed during the Diagnostic and written into the contract as a term. When the system operates inside that boundary and is wrong, the approval gate and the audit log are what contain it. When we have widened that boundary without agreement, that is our failure and it is a contractual matter, not a conversation.
It is tracked from the announcement rather than discovered from an error. The successor is scored against your fixed evaluation set, the behavioural differences are reported to you, thresholds are re-established, and cutover is scheduled with a rollback path. Providers publish these dates: OpenAI announced retirements from ChatGPT effective 13 February 2026, and Anthropic commits to at least 60 days notice on publicly released models.
That is the intended end state for most clients and it is designed for from the first month. The runbooks, evaluation set, infrastructure as code, monitoring configuration and report format are all yours and all documented as delivery proceeds. Handover is an administrative act rather than a project.
Monitoring is the input. The output is a decision each month and a change each quarter. If a report ever tells you everything is fine three months running with nothing changed, ask us what we are being paid for, because on a live AI system that is a sign nobody is looking hard enough.
Thirty minutes, no obligation. Tell us what is running, what you measure today and what you would notice if it degraded. If the honest answer is that you do not need this yet, we will say so.
Or write to [email protected].
AI agents and assistants
Inside the system, not beside it. The build that this page keeps working afterwards.
Service 06Cloud cost and FinOps
Rightsize first, commit second. Where the cost per completed task in the monthly report comes from.
DisclosureAI capability and data handling
Model providers, terms, residency, approval gates and rollback, written down before procurement asks.