An agent that checks a service every 10 minutes runs 4,320 times a month. If each pass is a tool-using investigation, at $0.56 to $1.26 per investigation according to an earlier Grafana Labs experiment, a single watcher costs roughly $2,400 to $5,400 a month. Grafana Assistant Watchers do the same job for a median of under $5 a month per watcher on Grafana Cloud's own infrastructure.
William Dumont of Grafana Labs explained how on November 9, 2026. The idea is easy to state and hard to execute: the model only steps in where judgment is needed, and only with as much model as each decision requires.
The problem: watching what alerts miss
Alert rules cover the failures someone predicted. Watchers are always-on agents for everything else: you describe in plain language what to watch (a service, a dependency, a fix from a recent incident), they learn what normal looks like, and they only speak up when behavior drifts, with a report and the evidence. For critical findings they can launch an Assistant investigation. They went GA at ObservabilityCON 2026.
The obstacle was cost. A capable, tool-using agent walking the stack on every pass doesn't scale to dozens of watchers per team.
The solution: three jobs, three executors
Grafana splits the work into three parts and gives each to the cheapest executor that can do it well:
| Job | Who does it | When |
|---|---|---|
| Decide what to watch and what normal is | Calibration agent (capable model, with tools) | Once, and on recalibration |
| Fetch data and compare it to the baseline | Code: queries, statistics, log patterns | Every run |
| Decide whether it matters and explain why | Small, fast model, one call, no tools | Every run |
Calibration is the expensive part, and it happens once. The agent maps where the telemetry lives, asks what it can't infer (for instance, whether 32 production clusters should be judged separately), mines existing alert rules and dashboards for thresholds and severities, samples each signal at several points over the past week to model daily and weekly cycles, and tests every check against live data before saving it. The output is a compact "calibration context": service overview, decisions made, deploy behavior, normal ranges and known noise. It takes a few minutes at most.
On every run, code does the arithmetic. For metrics, it compares against the same time yesterday and against a statistical baseline, then applies the calibrated thresholds, so the model gets verdicts rather than raw series. For logs it uses Loki's patterns API, which groups lines at ingestion; the watcher keeps a pattern inventory across runs, flags new patterns, catches known ones whose rate has doubled, and only fetches examples for the new ones. A deploy check lets it tell a post-rollout regression from a coincidence. Healthy checks collapse to one line each.
The model only decides. The digest goes to a single tool-free call to a small model, which returns a structured decision: OK, warning or critical, a short report, the issues it is tracking with their evidence, and whether to notify or investigate. To pick that model, Grafana replays real production decisions against the candidates; the current one missed no critical case and costs about half as much per decision as the one it replaced. Grafana doesn't say which model it is.
The numbers
At a list price of $2 per million tokens, 41 of the 43 internal watchers cost under $10 a month. The most expensive one, with 43 checks every 15 minutes, comes in a little over $20. No single run reaches a cent. Cost depends on frequency (from every 5 minutes to every 3 hours) and scope; calibration and investigations are billed separately.
Over the past few months, the internal watcher found more than 10 customer-facing bugs: a crashlooping Microsoft Teams integration, broken investigation reports, missing webhook credentials in a newly enabled region, Slack OAuth failures, and alerts that never reached Alertmanager. It also got one wrong: during a rollout it correctly flagged failures from an exhausted MySQL database but blamed the wrong cause; engineers later found an expensive query plan that likely contributed.
What to take away
The most useful lesson applies to any operations agent, Grafana's or your own: do the expensive reasoning once and reuse it on every run. An agent that rediscovers where the metrics live and what normal looks like on every pass pays for that discovery 4,320 times a month.
The second is about division of labor: whatever code does better than an LLM (querying, subtracting, comparing against a baseline) shouldn't reach the model as raw data. The model gets a digest that already says what is out of range, and its job shrinks to connecting signals. That is exactly where it earns its keep: a new log pattern on its own is usually noise, but a new pattern plus rising errors plus a recent deploy rarely is.
The third is about expectations. Grafana positions Watchers as a complement to alert rules, not a replacement, and separates detection from diagnosis: the watcher raises the flag, and people or an investigation find the cause. The MySQL episode shows why that line matters.
What doesn't carry over
The design leans on pieces not everyone has: Loki's patterns API does log grouping for free, and existing alerts and dashboards are the raw material for calibration. Without them, calibration costs more and is less reliable. The cost figures come from Grafana's internal fleet, at list price, with an unnamed model. And customer pricing isn't settled in practice yet: Watchers are free until December 1, 2026; after that they draw on the plan's monthly allowance for system-initiated usage and then bill on Assistant tokens.
Sources
- Run cost-effective, always-on agents to monitor your environment: the engineering behind Grafana Assistant Watchers, Grafana Labs, November 9, 2026.