Configuring Observability Dashboards with Claude: PromQL, Panels, and Alert Thresholds

1. Why Dashboards and Alerts So Often Fall Behind

Observability setups tend to grow organically alongside the services they monitor. A new endpoint gets a handful of metrics, someone quickly throws a panel together, and months later nobody can explain why the latency alert fires at exactly 800 milliseconds. The tooling is rarely the problem, Grafana and Prometheus are mature and well documented, the real gap is having enough time to think dashboards and thresholds through properly before the next sprint pulls attention elsewhere.

This is exactly where using Claude as a fast sparring partner pays off. It delivers panel drafts, reviews existing PromQL expressions for common mistakes, and proposes thresholds based on historical values you feed it. The actual decision about which signals matter for a given service still belongs to the team, but the path to that decision gets much shorter because the tedious first draft no longer has to be typed by hand.

2. Generating Grafana Dashboard JSON from a Description

The most obvious entry point is giving Claude a text description of the panels you want and having it produce valid dashboard JSON that can be loaded directly via provisioning or manual import into Grafana. What matters is supplying enough context: which data source is used, what the metrics are actually called, and which Grafana schema version is in play, so the generated panels work without rework and do not reference outdated field names.

In practice this works best iteratively. Generate a first draft with the most important panels, import the JSON locally into a test instance, then give concrete feedback about individual panels, for example that a unit is wrong or a legend is missing. Claude then adjusts exactly that part of the JSON without touching the rest of the dashboard, which is noticeably faster than a full regeneration on larger dashboards with many rows.

{
  "title": "Checkout Service Overview",
  "panels": [
    {
      "type": "timeseries",
      "title": "Request Latency p95",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service=\"checkout\"}[5m])) by (le))",
          "legendFormat": "p95 latency"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "s" }
      }
    }
  ]
}

3. Having Claude Review and Improve PromQL Queries

A second, often underused, application is pure query review. Many PromQL expressions found in dashboards that grew over time use rate() with a far too short window, forget to aggregate over the le label with by (le) before histogram_quantile(), or mix counter and gauge metrics in the same formula, producing misleading curves. Claude reliably spots these patterns once you paste the existing queries and ask for a short explanation of the underlying metric types.

This is especially valuable when inheriting someone else's dashboards, for example after a team handover or when migrating a community dashboard from the Grafana marketplace. Instead of manually tracing every single query, you have Claude produce a prioritized list of concerns ranked by likelihood of an actual bug, then work through that list deliberately rather than trusting blindly that an imported dashboard is already correct.

# Before: window far too short, no aggregation by le
rate(http_request_duration_seconds_bucket[30s])

# After Claude's review: correct window
# and correct aggregation before histogram_quantile
histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

4. Thinking Through Alert Thresholds Instead of Guessing

A static threshold such as a fixed 500 milliseconds for latency is almost always a compromise between gut feeling and time pressure. It is far more useful to give Claude the actual distribution of recent weeks, for example an exported CSV with hourly percentile values, and have it derive a threshold grounded in the real steady state rather than a round number. A latency alert set at the 99th percentile of the last thirty days plus a safety margin fires far less often for no reason than a guessed fixed value.

It also helps to explicitly ask Claude about the trade off between sensitivity and reaction time. A threshold that is too tight generates constant false alarms during normal load spikes, one that is too loose leaves real problems unnoticed for a long time. Claude can suggest different threshold styles for different metric types, for example percentile based thresholds for latency and trend based forecasts for resource usage, instead of applying the same rigid percentage rule everywhere.

groups:
  - name: checkout-latency
    rules:
      - alert: CheckoutLatencyHigh
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket{service="checkout"}[5m])) by (le)
          ) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Checkout p95 latency above historical baseline"

5. Avoiding Alert Fatigue: Fewer but More Relevant Alerts

Alert fatigue sets in when a team gets bombarded with alarms for weeks, most of them harmless, until eventually the real incident gets lost in the noise and nobody reacts right away. A proven step is to hand Claude the complete list of active alert rules and ask it to flag anomalies: duplicate alerts describing the same root cause, alerts without a clear action in the annotation, or rules that have not fired in months and are probably misconfigured.

That analysis usually yields a concrete consolidation list, for example merging several separate error rate alerts per endpoint into a single service grouped alert with label based breakdown. It is important to sanity check every consolidation with the team, since Claude does not automatically know the organizational reasons behind some seemingly redundant rule, for example when two teams deliberately need separate alerts for the same service because they own different areas of responsibility.

6. Designing Multi Window Multi Burn Rate Alerts with Claude

A pattern from the Google SRE workbook that remains underused in many teams is multi window multi burn rate alerting for SLO error budgets. Instead of a single rigid threshold, you combine a short and a long window: only when both windows simultaneously show an elevated burn rate does the alert actually fire. This drastically reduces false alarms from short lived spikes without slowing down reaction time on real, sustained problems.

The formulas are not complex but easy to get wrong when writing them from scratch, especially the correct burn rate calculation from the error budget and the SLO target. Claude works well here, deriving matching Prometheus rules for multiple windows and severities directly from a given SLO definition, for example 99.9 percent success rate over 30 days, including the common combination of a 5 minute and a 1 hour window for critical alerts plus a 6 hour and 3 day window for less urgent warnings.

- alert: ErrorBudgetBurnRateCritical
  expr: |
    (
      sum(rate(http_requests_total{status=~"5..",service="checkout"}[5m]))
      /
      sum(rate(http_requests_total{service="checkout"}[5m]))
    ) > (14.4 * 0.001)
    and
    (
      sum(rate(http_requests_total{status=~"5..",service="checkout"}[1h]))
      /
      sum(rate(http_requests_total{service="checkout"}[1h]))
    ) > (14.4 * 0.001)
  labels:
    severity: critical

7. Deriving Dashboard Variants for Different Audiences

A technically deep SRE dashboard with twenty panels is not much use to an executive who wants a quick daily glance at system health. Instead of rebuilding a second dashboard entirely by hand, you can ask Claude to derive a reduced variant from the existing detailed dashboard JSON, keeping only the four or five most meaningful panels and rewording titles without internal jargon.

The same principle works in reverse: deriving a more detailed debugging variant from a simple overview dashboard for the next incident, adding panels for individual instances, database connection pools, or queue lengths. Because the underlying JSON structure stays consistent, both variants can be maintained in parallel without manually porting every change from one version to the other.

8. Using Claude Code Directly in the Grafana Provisioning Repository

Teams that manage dashboards and alerts as code in a git repository with Grafana provisioning benefit from letting Claude Code work directly inside that repository, instead of copying JSON snippets back and forth between a chat window and the codebase. Claude Code then sees the existing dashboards, can pick up the established style, and names new panels consistently with the existing ones instead of starting from zero every time.

The workflow stays deliberately reviewable: Claude Code edits files in the working directory, the team sees the full diff through the usual version control system before anything gets committed or deployed to the production Grafana instance. Especially for alert rules, where a misconfiguration in the worst case means nobody gets notified at all, that code review step is not negotiable.

cd observability-as-code
claude "Update alerts/checkout.yaml: add a multi window
burn rate rule for the 99.9 percent SLO, in the same
style as alerts/payment.yaml"

git diff alerts/checkout.yaml
# Review, then commit
git add alerts/checkout.yaml
git commit -m "Add burn rate alert for checkout SLO"

9. Limits: Claude Does Not Automatically Know Your System

Claude knows neither your system's actual metric names nor the real distribution of your latency values unless that information is explicitly supplied. Without concrete context, the model produces plausible looking but potentially wrong metric names or thresholds that are not tied to any real observation. Every generated query and every suggested threshold therefore needs to be checked against real data before it goes live.

A staged rollout has proven effective: run new alert rules in pure observation mode without notifications first, watch their firing behavior against real production data for a week, and only then enable actual paging. Claude provides a solid, well reasoned starting point and speeds up creation considerably, but the responsibility for the final sign off and calibration stays with the team, who bears the consequences of a false alarm or a missed real incident.

10. Summary

11. FAQ: Observability Dashboards with Claude: FAQ